Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
2026-05-01T13:57:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T13:57:13Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260501 13:57:13.120054 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.190:45953
--webserver_interface=127.1.16.190
--webserver_port=0
--builtin_ntp_servers=127.1.16.148:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260501 13:57:13.417732 1099 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:57:13.418334 1099 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:57:13.418779 1099 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 13:57:13.452263 1099 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 13:57:13.452602 1099 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:57:13.452853 1099 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 13:57:13.453092 1099 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 13:57:13.489385 1099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.190:45953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.16.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:57:13.490756 1099 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:57:13.492442 1099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 13:57:13.504940 1105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:13.505272 1106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:14.633147 1108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:14.634929 1107 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1124 milliseconds
I20260501 13:57:14.635073 1099 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260501 13:57:14.636329 1099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:57:14.638875 1099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:57:14.640230 1099 hybrid_clock.cc:648] HybridClock initialized: now 1777643834640172 us; error 64 us; skew 500 ppm
I20260501 13:57:14.641009 1099 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:57:14.648094 1099 webserver.cc:492] Webserver started at http://127.1.16.190:44035/ using document root <none> and password file <none>
I20260501 13:57:14.649011 1099 fs_manager.cc:362] Metadata directory not provided
I20260501 13:57:14.649217 1099 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:57:14.649631 1099 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:57:14.653970 1099 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/data/instance:
uuid: "f04a112d3be549d98b4ee77dbefacb7b"
format_stamp: "Formatted at 2026-05-01 13:57:14 on dist-test-slave-nqxd"
I20260501 13:57:14.655102 1099 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/wal/instance:
uuid: "f04a112d3be549d98b4ee77dbefacb7b"
format_stamp: "Formatted at 2026-05-01 13:57:14 on dist-test-slave-nqxd"
I20260501 13:57:14.662876 1099 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260501 13:57:14.668933 1115 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:14.670054 1099 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260501 13:57:14.670393 1099 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/wal
uuid: "f04a112d3be549d98b4ee77dbefacb7b"
format_stamp: "Formatted at 2026-05-01 13:57:14 on dist-test-slave-nqxd"
I20260501 13:57:14.670761 1099 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:57:14.724979 1099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:57:14.726981 1099 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:57:14.727433 1099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:57:14.803851 1099 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.190:45953
I20260501 13:57:14.803926 1166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.190:45953 every 8 connection(s)
I20260501 13:57:14.806528 1099 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/data/info.pb
I20260501 13:57:14.809888 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 1099
I20260501 13:57:14.810400 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-0/wal/instance
I20260501 13:57:14.814181 1167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:14.815668 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.189:45897
--webserver_interface=127.1.16.189
--webserver_port=0
--builtin_ntp_servers=127.1.16.148:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260501 13:57:14.826579 1167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:57:14.850965 1167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } has no permanent_uuid. Determining permanent_uuid...
W20260501 13:57:14.853603 1117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.16.189:45897: connect: Connection refused (error 111)
W20260501 13:57:14.857329 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:45897: Network error: Client connection negotiation failed: client connection to 127.1.16.189:45897: connect: Connection refused (error 111)
I20260501 13:57:14.906874 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } attempt: 1
W20260501 13:57:14.910565 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:45897: Network error: Client connection negotiation failed: client connection to 127.1.16.189:45897: connect: Connection refused (error 111)
I20260501 13:57:14.979050 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } attempt: 2
W20260501 13:57:14.982797 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:45897: Network error: Client connection negotiation failed: client connection to 127.1.16.189:45897: connect: Connection refused (error 111)
I20260501 13:57:15.074244 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } attempt: 3
W20260501 13:57:15.077929 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:45897: Network error: Client connection negotiation failed: client connection to 127.1.16.189:45897: connect: Connection refused (error 111)
W20260501 13:57:15.116272 1169 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:57:15.116770 1169 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:57:15.117126 1169 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 13:57:15.148761 1169 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 13:57:15.149035 1169 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:57:15.149241 1169 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 13:57:15.149436 1169 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 13:57:15.185284 1169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.189:45897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.16.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:57:15.186599 1169 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:57:15.188144 1169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 13:57:15.200277 1178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:57:15.221379 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } attempt: 4
W20260501 13:57:15.227478 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:45897: Network error: Client connection negotiation failed: client connection to 127.1.16.189:45897: connect: Connection refused (error 111)
I20260501 13:57:15.526978 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } attempt: 5
W20260501 13:57:15.533077 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:45897: Network error: Client connection negotiation failed: client connection to 127.1.16.189:45897: connect: Connection refused (error 111)
I20260501 13:57:16.080575 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } attempt: 6
W20260501 13:57:16.105808 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:45897: Network error: Client connection negotiation failed: client connection to 127.1.16.189:45897: connect: Connection refused (error 111)
W20260501 13:57:15.200326 1179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:16.503039 1180 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1297 milliseconds
W20260501 13:57:16.503940 1169 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.305s user 0.494s sys 0.809s
W20260501 13:57:16.504256 1169 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.305s user 0.494s sys 0.809s
W20260501 13:57:16.504161 1181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:57:16.504488 1169 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260501 13:57:16.505540 1169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:57:16.508538 1169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:57:16.509938 1169 hybrid_clock.cc:648] HybridClock initialized: now 1777643836509901 us; error 45 us; skew 500 ppm
I20260501 13:57:16.510782 1169 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:57:16.519624 1169 webserver.cc:492] Webserver started at http://127.1.16.189:36903/ using document root <none> and password file <none>
I20260501 13:57:16.520525 1169 fs_manager.cc:362] Metadata directory not provided
I20260501 13:57:16.520761 1169 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:57:16.521202 1169 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:57:16.525665 1169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/data/instance:
uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51"
format_stamp: "Formatted at 2026-05-01 13:57:16 on dist-test-slave-nqxd"
I20260501 13:57:16.526806 1169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/wal/instance:
uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51"
format_stamp: "Formatted at 2026-05-01 13:57:16 on dist-test-slave-nqxd"
I20260501 13:57:16.535104 1169 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.002s
I20260501 13:57:16.541252 1190 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:16.542518 1169 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20260501 13:57:16.542879 1169 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/wal
uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51"
format_stamp: "Formatted at 2026-05-01 13:57:16 on dist-test-slave-nqxd"
I20260501 13:57:16.543233 1169 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:57:16.629994 1169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:57:16.632056 1169 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:57:16.632517 1169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:57:16.709872 1169 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.189:45897
I20260501 13:57:16.709928 1241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.189:45897 every 8 connection(s)
I20260501 13:57:16.712491 1169 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/data/info.pb
I20260501 13:57:16.717078 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 1169
I20260501 13:57:16.717556 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-1/wal/instance
I20260501 13:57:16.719964 1242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:16.720801 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.188:38815
--webserver_interface=127.1.16.188
--webserver_port=0
--builtin_ntp_servers=127.1.16.148:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260501 13:57:16.732575 1242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:57:16.754427 1242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:57:16.768613 1242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } has no permanent_uuid. Determining permanent_uuid...
W20260501 13:57:16.771062 1194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
W20260501 13:57:16.775458 1242 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:16.824944 1242 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 1
W20260501 13:57:16.828824 1242 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:16.897192 1242 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 2
W20260501 13:57:16.901192 1242 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:16.992625 1242 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 3
W20260501 13:57:16.996924 1242 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
W20260501 13:57:17.031963 1244 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:57:17.032539 1244 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:57:17.032940 1244 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 13:57:17.065445 1244 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 13:57:17.065750 1244 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:57:17.066000 1244 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 13:57:17.066226 1244 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 13:57:17.102980 1244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.188:38815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.16.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:57:17.104245 1244 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:57:17.105816 1244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 13:57:17.118431 1254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:57:17.140374 1242 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 4
W20260501 13:57:17.145408 1242 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:17.166301 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } attempt: 7
I20260501 13:57:17.181582 1167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } has no permanent_uuid. Determining permanent_uuid...
W20260501 13:57:17.189306 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:17.247895 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 1
W20260501 13:57:17.253171 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:17.334654 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 2
W20260501 13:57:17.339772 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:17.425311 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 3
W20260501 13:57:17.431254 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:17.444871 1242 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 5
W20260501 13:57:17.450009 1242 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:17.571766 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 4
W20260501 13:57:17.577791 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:17.861471 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 5
W20260501 13:57:17.866917 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:17.997538 1242 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 6
W20260501 13:57:18.006640 1242 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
W20260501 13:57:17.120419 1255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:18.310585 1257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:18.312565 1256 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1190 milliseconds
I20260501 13:57:18.312708 1244 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260501 13:57:18.313928 1244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:57:18.316450 1244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:57:18.317777 1244 hybrid_clock.cc:648] HybridClock initialized: now 1777643838317762 us; error 46 us; skew 500 ppm
I20260501 13:57:18.318637 1244 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:57:18.325474 1244 webserver.cc:492] Webserver started at http://127.1.16.188:41671/ using document root <none> and password file <none>
I20260501 13:57:18.326750 1244 fs_manager.cc:362] Metadata directory not provided
I20260501 13:57:18.326977 1244 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:57:18.327394 1244 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:57:18.332223 1244 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data/instance:
uuid: "50c0932256e74399a537670bb424a1b7"
format_stamp: "Formatted at 2026-05-01 13:57:18 on dist-test-slave-nqxd"
I20260501 13:57:18.333338 1244 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal/instance:
uuid: "50c0932256e74399a537670bb424a1b7"
format_stamp: "Formatted at 2026-05-01 13:57:18 on dist-test-slave-nqxd"
I20260501 13:57:18.341295 1244 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260501 13:57:18.348968 1268 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:18.350042 1244 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260501 13:57:18.350348 1244 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal
uuid: "50c0932256e74399a537670bb424a1b7"
format_stamp: "Formatted at 2026-05-01 13:57:18 on dist-test-slave-nqxd"
I20260501 13:57:18.350687 1244 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:57:18.398574 1244 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:57:18.400038 1244 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:57:18.400440 1244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:57:18.419521 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 6
W20260501 13:57:18.428476 1167 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38815: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:18.480569 1244 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.188:38815
I20260501 13:57:18.480633 1321 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.188:38815 every 8 connection(s)
I20260501 13:57:18.483219 1244 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
I20260501 13:57:18.489302 1322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:18.490247 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 1244
I20260501 13:57:18.490674 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal/instance
I20260501 13:57:18.505847 1322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:57:18.531435 1322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:57:18.543265 1322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:57:18.560295 1322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: Bootstrap starting.
I20260501 13:57:18.568260 1322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:18.571010 1322 log.cc:826] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: Log is configured to *not* fsync() on all Append() calls
I20260501 13:57:18.575263 1322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: No bootstrap required, opened a new log
I20260501 13:57:18.596799 1322 raft_consensus.cc:359] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:18.597437 1322 raft_consensus.cc:385] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:18.597715 1322 raft_consensus.cc:740] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c0932256e74399a537670bb424a1b7, State: Initialized, Role: FOLLOWER
I20260501 13:57:18.598419 1322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:18.601426 1333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:18.602221 1333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:18.603264 1322 sys_catalog.cc:565] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: configured and running, proceeding with master startup.
W20260501 13:57:18.628571 1344 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:57:18.631484 1344 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:57:18.624174 1244 master_runner.cc:186] Error getting master registration for 127.1.16.190:45953: OK, instance_id { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" instance_seqno: 1777643834781938 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260501 13:57:18.654917 1244 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260501 13:57:18.655835 1274 tablet.cc:2404] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:57:19.028895 1333 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:19.029500 1333 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:19.031522 1333 leader_election.cc:290] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897)
I20260501 13:57:19.067063 1242 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 7
I20260501 13:57:19.092396 1242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51: Bootstrap starting.
I20260501 13:57:19.100208 1242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:19.103145 1242 log.cc:826] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51: Log is configured to *not* fsync() on all Append() calls
I20260501 13:57:19.108731 1242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51: No bootstrap required, opened a new log
I20260501 13:57:19.136900 1242 raft_consensus.cc:359] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:19.137713 1242 raft_consensus.cc:385] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:19.138087 1242 raft_consensus.cc:740] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18ed9dc598e4ccaaf8633eaa45a2f51, State: Initialized, Role: FOLLOWER
I20260501 13:57:19.139027 1242 consensus_queue.cc:260] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:19.142338 1349 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:19.143074 1349 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:19.143889 1242 sys_catalog.cc:565] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 13:57:19.144050 1217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" is_pre_election: true
I20260501 13:57:19.144862 1217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50c0932256e74399a537670bb424a1b7 in term 0.
I20260501 13:57:19.146118 1270 leader_election.cc:304] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50c0932256e74399a537670bb424a1b7, d18ed9dc598e4ccaaf8633eaa45a2f51; no voters:
I20260501 13:57:19.147308 1333 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:19.147740 1333 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:19.148048 1333 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:19.175477 1360 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:57:19.175954 1360 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:57:19.173475 1169 master_runner.cc:186] Error getting master registration for 127.1.16.190:45953: OK, instance_id { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" instance_seqno: 1777643834781938 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260501 13:57:19.202839 1169 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260501 13:57:19.206904 1333 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:19.209364 1333 leader_election.cc:290] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 1 election: Requested vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897)
I20260501 13:57:19.211670 1217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51"
I20260501 13:57:19.212260 1217 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:19.309049 1217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50c0932256e74399a537670bb424a1b7 in term 1.
I20260501 13:57:19.313266 1270 leader_election.cc:304] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50c0932256e74399a537670bb424a1b7, d18ed9dc598e4ccaaf8633eaa45a2f51; no voters:
I20260501 13:57:19.314122 1333 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:19.316625 1333 raft_consensus.cc:697] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 1 LEADER]: Becoming Leader. State: Replica: 50c0932256e74399a537670bb424a1b7, State: Running, Role: LEADER
I20260501 13:57:19.317623 1333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:19.325099 1362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50c0932256e74399a537670bb424a1b7. Latest consensus state: current_term: 1 leader_uuid: "50c0932256e74399a537670bb424a1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:19.326037 1362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: This master's current role is: LEADER
I20260501 13:57:19.330664 1364 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 13:57:19.343143 1364 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 13:57:19.365861 1217 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Refusing update from remote peer 50c0932256e74399a537670bb424a1b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:57:19.367456 1333 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:19.391363 1196 tablet.cc:2404] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:57:19.398741 1365 mvcc.cc:204] Tried to move back new op lower bound from 7281229166016368640 to 7281229165854728192. Current Snapshot: MvccSnapshot[applied={T|T < 7281229166016368640}]
I20260501 13:57:19.396176 1349 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50c0932256e74399a537670bb424a1b7. Latest consensus state: current_term: 1 leader_uuid: "50c0932256e74399a537670bb424a1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:19.402683 1349 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:19.412926 1362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50c0932256e74399a537670bb424a1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:19.413965 1362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: This master's current role is: LEADER
I20260501 13:57:19.416492 1364 catalog_manager.cc:1357] Generated new cluster ID: 5d0126731b4a4ae2a770b98ff0c83307
I20260501 13:57:19.416788 1364 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 13:57:19.454140 1349 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50c0932256e74399a537670bb424a1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:19.454871 1349 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:19.462195 1167 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } attempt: 7
I20260501 13:57:19.477388 1364 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 13:57:19.479493 1364 catalog_manager.cc:1514] Loading token signing keys...
I20260501 13:57:19.529639 1364 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: Generated new TSK 0
I20260501 13:57:19.530757 1364 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 13:57:19.531801 1167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Bootstrap starting.
I20260501 13:57:19.547423 1167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:19.550249 1167 log.cc:826] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Log is configured to *not* fsync() on all Append() calls
I20260501 13:57:19.561455 1167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: No bootstrap required, opened a new log
I20260501 13:57:19.590092 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.16.129:0
--local_ip_for_outbound_sockets=127.1.16.129
--webserver_interface=127.1.16.129
--webserver_port=0
--tserver_master_addrs=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--builtin_ntp_servers=127.1.16.148:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260501 13:57:19.614353 1167 raft_consensus.cc:359] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:19.615196 1167 raft_consensus.cc:385] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:19.615494 1167 raft_consensus.cc:740] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04a112d3be549d98b4ee77dbefacb7b, State: Initialized, Role: FOLLOWER
I20260501 13:57:19.616336 1167 consensus_queue.cc:260] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:19.638309 1167 sys_catalog.cc:565] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [sys.catalog]: configured and running, proceeding with master startup.
I20260501 13:57:19.644068 1140 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:19.641700 1141 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b"
I20260501 13:57:19.642361 1142 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
W20260501 13:57:19.653656 1142 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41688 (request call id 2) took 621 ms (client timeout 744 ms). Trace:
W20260501 13:57:19.654037 1142 rpcz_store.cc:269] 0501 13:57:19.032019 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:19.032147 (+ 128us) service_pool.cc:224] Handling call
0501 13:57:19.653577 (+621430us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260501 13:57:19.643605 1374 sys_catalog.cc:455] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:19.663097 1374 sys_catalog.cc:458] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:19.717962 1140 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Refusing update from remote peer 50c0932256e74399a537670bb424a1b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:57:19.719842 1333 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:19.738366 1121 tablet.cc:2404] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:57:19.785946 1374 sys_catalog.cc:455] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50c0932256e74399a537670bb424a1b7. Latest consensus state: current_term: 1 leader_uuid: "50c0932256e74399a537670bb424a1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:19.786041 1384 sys_catalog.cc:455] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50c0932256e74399a537670bb424a1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:19.786718 1374 sys_catalog.cc:458] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:19.786878 1384 sys_catalog.cc:458] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:19.858742 1390 catalog_manager.cc:1269] Loaded cluster ID: 5d0126731b4a4ae2a770b98ff0c83307
I20260501 13:57:19.859026 1390 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: loading cluster ID for follower catalog manager: success
I20260501 13:57:19.879200 1390 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: acquiring CA information for follower catalog manager: success
I20260501 13:57:19.884857 1390 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 13:57:20.062731 1392 raft_consensus.cc:493] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:20.063475 1392 raft_consensus.cc:515] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:20.066742 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
I20260501 13:57:20.068032 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7" is_pre_election: true
I20260501 13:57:20.070165 1194 leader_election.cc:304] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d18ed9dc598e4ccaaf8633eaa45a2f51; no voters: 50c0932256e74399a537670bb424a1b7, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:20.076004 1392 leader_election.cc:290] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:20.081841 1392 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:57:20.166623 1394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50c0932256e74399a537670bb424a1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:20.169600 1394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: This master's current role is: LEADER
I20260501 13:57:20.181619 1360 catalog_manager.cc:1269] Loaded cluster ID: 5d0126731b4a4ae2a770b98ff0c83307
I20260501 13:57:20.181974 1360 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51: loading cluster ID for follower catalog manager: success
I20260501 13:57:20.192414 1360 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51: acquiring CA information for follower catalog manager: success
I20260501 13:57:20.197851 1360 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260501 13:57:20.234666 1373 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:57:20.235334 1373 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:57:20.236217 1373 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 13:57:20.296948 1373 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:57:20.298454 1373 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.16.129
I20260501 13:57:20.363387 1373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.16.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.16.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.16.129
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:57:20.365172 1373 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:57:20.367429 1373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 13:57:20.396584 1402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:57:20.602262 1408 raft_consensus.cc:493] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:20.612253 1408 raft_consensus.cc:515] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:20.614995 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
I20260501 13:57:20.617770 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7" is_pre_election: true
I20260501 13:57:20.619146 1194 leader_election.cc:304] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d18ed9dc598e4ccaaf8633eaa45a2f51; no voters: 50c0932256e74399a537670bb424a1b7, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:20.624802 1408 leader_election.cc:290] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:20.625679 1408 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:57:21.625267 1410 raft_consensus.cc:493] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:21.630770 1410 raft_consensus.cc:515] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:21.634131 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
I20260501 13:57:21.637727 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7" is_pre_election: true
I20260501 13:57:21.639274 1194 leader_election.cc:304] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d18ed9dc598e4ccaaf8633eaa45a2f51; no voters: 50c0932256e74399a537670bb424a1b7, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:21.644973 1410 leader_election.cc:290] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:21.646124 1410 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:57:21.789665 1401 debug-util.cc:398] Leaking SignalData structure 0x7b0800028ca0 after lost signal to thread 1373
W20260501 13:57:21.940896 1373 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.533s user 0.279s sys 0.428s
W20260501 13:57:20.408886 1403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:21.941313 1373 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.533s user 0.279s sys 0.428s
W20260501 13:57:21.943866 1406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:21.950048 1404 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1541 milliseconds
I20260501 13:57:21.950083 1373 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260501 13:57:21.952540 1373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:57:21.955395 1373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:57:21.956959 1373 hybrid_clock.cc:648] HybridClock initialized: now 1777643841956916 us; error 51 us; skew 500 ppm
I20260501 13:57:21.958120 1373 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:57:21.966318 1373 webserver.cc:492] Webserver started at http://127.1.16.129:46343/ using document root <none> and password file <none>
I20260501 13:57:21.967576 1373 fs_manager.cc:362] Metadata directory not provided
I20260501 13:57:21.967868 1373 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:57:21.968429 1373 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:57:21.974696 1373 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/data/instance:
uuid: "993b9ba18d8547f8af43167e727a6a24"
format_stamp: "Formatted at 2026-05-01 13:57:21 on dist-test-slave-nqxd"
I20260501 13:57:21.976231 1373 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/instance:
uuid: "993b9ba18d8547f8af43167e727a6a24"
format_stamp: "Formatted at 2026-05-01 13:57:21 on dist-test-slave-nqxd"
I20260501 13:57:21.987440 1415 raft_consensus.cc:493] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:21.988225 1415 raft_consensus.cc:515] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:21.991773 1217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" is_pre_election: true
I20260501 13:57:21.992322 1415 leader_election.cc:290] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:21.993744 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7" is_pre_election: true
I20260501 13:57:21.996196 1119 leader_election.cc:304] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f04a112d3be549d98b4ee77dbefacb7b; no voters: 50c0932256e74399a537670bb424a1b7, d18ed9dc598e4ccaaf8633eaa45a2f51
I20260501 13:57:21.997589 1415 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:57:22.000109 1373 fs_manager.cc:696] Time spent creating directory manager: real 0.023s user 0.009s sys 0.015s
I20260501 13:57:22.007021 1417 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:22.008158 1373 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260501 13:57:22.008490 1373 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal
uuid: "993b9ba18d8547f8af43167e727a6a24"
format_stamp: "Formatted at 2026-05-01 13:57:21 on dist-test-slave-nqxd"
I20260501 13:57:22.008836 1373 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:57:22.057653 1373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:57:22.059211 1373 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:57:22.059651 1373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:57:22.063023 1373 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 13:57:22.068104 1373 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 13:57:22.068333 1373 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:22.068598 1373 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 13:57:22.068763 1373 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:22.231248 1496 raft_consensus.cc:493] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:22.232571 1496 raft_consensus.cc:515] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:22.236644 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
I20260501 13:57:22.240446 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7" is_pre_election: true
I20260501 13:57:22.242290 1194 leader_election.cc:304] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d18ed9dc598e4ccaaf8633eaa45a2f51; no voters: 50c0932256e74399a537670bb424a1b7, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:22.247803 1496 leader_election.cc:290] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:22.248282 1498 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:57:22.319099 1373 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.129:38663
I20260501 13:57:22.319183 1531 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.129:38663 every 8 connection(s)
I20260501 13:57:22.321604 1373 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/data/info.pb
I20260501 13:57:22.326958 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 1373
I20260501 13:57:22.327457 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/instance
I20260501 13:57:22.385102 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.16.130:0
--local_ip_for_outbound_sockets=127.1.16.130
--webserver_interface=127.1.16.130
--webserver_port=0
--tserver_master_addrs=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--builtin_ntp_servers=127.1.16.148:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260501 13:57:22.394078 1533 heartbeater.cc:344] Connected to a master server at 127.1.16.189:45897
I20260501 13:57:22.394610 1533 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:22.396443 1533 heartbeater.cc:507] Master 127.1.16.189:45897 requested a full tablet report, sending...
I20260501 13:57:22.400161 1207 ts_manager.cc:194] Registered new tserver with Master: 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:22.422169 1535 heartbeater.cc:344] Connected to a master server at 127.1.16.190:45953
I20260501 13:57:22.422542 1535 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:22.423413 1535 heartbeater.cc:507] Master 127.1.16.190:45953 requested a full tablet report, sending...
I20260501 13:57:22.427390 1132 ts_manager.cc:194] Registered new tserver with Master: 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:22.432281 1532 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:22.432643 1532 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:22.433557 1532 heartbeater.cc:507] Master 127.1.16.188:38815 requested a full tablet report, sending...
I20260501 13:57:22.437729 1287 ts_manager.cc:194] Registered new tserver with Master: 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:22.440488 1287 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.16.129:54785
I20260501 13:57:22.556123 1543 raft_consensus.cc:493] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:22.556733 1543 raft_consensus.cc:515] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:22.559391 1543 leader_election.cc:290] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:22.559964 1217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" is_pre_election: true
I20260501 13:57:22.560454 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7" is_pre_election: true
I20260501 13:57:22.562036 1117 leader_election.cc:304] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f04a112d3be549d98b4ee77dbefacb7b; no voters: 50c0932256e74399a537670bb424a1b7, d18ed9dc598e4ccaaf8633eaa45a2f51
I20260501 13:57:22.563189 1543 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:57:22.790558 1542 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:57:22.791080 1542 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:57:22.791620 1542 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 13:57:22.844201 1542 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:57:22.845757 1542 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.16.130
I20260501 13:57:22.884588 1542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.16.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.16.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.16.130
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:57:22.885897 1542 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:57:22.887499 1542 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 13:57:22.901918 1549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:57:22.944487 1532 heartbeater.cc:499] Master 127.1.16.188:38815 was elected leader, sending a full tablet report...
I20260501 13:57:23.228332 1556 raft_consensus.cc:493] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:23.228928 1556 raft_consensus.cc:515] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:23.231024 1556 leader_election.cc:290] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:23.231899 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
I20260501 13:57:23.233072 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7" is_pre_election: true
I20260501 13:57:23.234659 1194 leader_election.cc:304] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d18ed9dc598e4ccaaf8633eaa45a2f51; no voters: 50c0932256e74399a537670bb424a1b7, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:23.235524 1556 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:57:22.908517 1550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:22.911392 1552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:24.023770 1551 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260501 13:57:24.023852 1542 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260501 13:57:24.027920 1542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:57:24.030383 1542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:57:24.031821 1542 hybrid_clock.cc:648] HybridClock initialized: now 1777643844031783 us; error 55 us; skew 500 ppm
I20260501 13:57:24.032610 1542 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:57:24.042197 1542 webserver.cc:492] Webserver started at http://127.1.16.130:36385/ using document root <none> and password file <none>
I20260501 13:57:24.043169 1542 fs_manager.cc:362] Metadata directory not provided
I20260501 13:57:24.043371 1542 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:57:24.043815 1542 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:57:24.048005 1542 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/data/instance:
uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
format_stamp: "Formatted at 2026-05-01 13:57:24 on dist-test-slave-nqxd"
I20260501 13:57:24.049052 1542 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/instance:
uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
format_stamp: "Formatted at 2026-05-01 13:57:24 on dist-test-slave-nqxd"
I20260501 13:57:24.055902 1542 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260501 13:57:24.061288 1562 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:24.062301 1542 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260501 13:57:24.062633 1542 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal
uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
format_stamp: "Formatted at 2026-05-01 13:57:24 on dist-test-slave-nqxd"
I20260501 13:57:24.062948 1542 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:57:24.152297 1542 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:57:24.153760 1542 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:57:24.154177 1542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:57:24.156847 1542 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 13:57:24.160943 1542 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 13:57:24.161144 1542 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:24.161377 1542 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 13:57:24.161588 1542 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:24.307875 1542 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.130:36735
I20260501 13:57:24.307983 1674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.130:36735 every 8 connection(s)
I20260501 13:57:24.310316 1542 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/data/info.pb
I20260501 13:57:24.320060 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 1542
I20260501 13:57:24.320698 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/instance
I20260501 13:57:24.334154 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.16.131:0
--local_ip_for_outbound_sockets=127.1.16.131
--webserver_interface=127.1.16.131
--webserver_port=0
--tserver_master_addrs=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--builtin_ntp_servers=127.1.16.148:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260501 13:57:24.372709 1675 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:24.373225 1675 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:24.375182 1675 heartbeater.cc:507] Master 127.1.16.188:38815 requested a full tablet report, sending...
I20260501 13:57:24.376770 1678 heartbeater.cc:344] Connected to a master server at 127.1.16.190:45953
I20260501 13:57:24.377113 1678 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:24.378122 1678 heartbeater.cc:507] Master 127.1.16.190:45953 requested a full tablet report, sending...
I20260501 13:57:24.378733 1287 ts_manager.cc:194] Registered new tserver with Master: 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:24.379495 1676 heartbeater.cc:344] Connected to a master server at 127.1.16.189:45897
I20260501 13:57:24.379835 1676 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:24.380146 1287 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.16.130:55415
I20260501 13:57:24.380636 1676 heartbeater.cc:507] Master 127.1.16.189:45897 requested a full tablet report, sending...
I20260501 13:57:24.381055 1132 ts_manager.cc:194] Registered new tserver with Master: 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:24.384239 1207 ts_manager.cc:194] Registered new tserver with Master: 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
W20260501 13:57:24.669268 1685 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:57:24.669783 1685 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:57:24.670322 1685 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260501 13:57:24.680727 1690 raft_consensus.cc:493] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:24.681111 1690 raft_consensus.cc:515] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:24.682628 1690 leader_election.cc:290] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:24.683661 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7" is_pre_election: true
I20260501 13:57:24.683656 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
I20260501 13:57:24.685109 1193 leader_election.cc:304] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d18ed9dc598e4ccaaf8633eaa45a2f51; no voters: 50c0932256e74399a537670bb424a1b7, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:24.685693 1690 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:57:24.705020 1685 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:57:24.705982 1685 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.16.131
I20260501 13:57:24.760705 1685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.16.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.16.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.16.131
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:57:24.762193 1685 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:57:24.763780 1685 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 13:57:24.776623 1693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:57:24.884793 1675 heartbeater.cc:499] Master 127.1.16.188:38815 was elected leader, sending a full tablet report...
I20260501 13:57:25.257768 1701 raft_consensus.cc:493] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:25.257812 1700 raft_consensus.cc:493] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:25.258625 1701 raft_consensus.cc:515] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:25.258791 1700 raft_consensus.cc:515] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:25.262243 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
I20260501 13:57:25.262503 1217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" is_pre_election: true
I20260501 13:57:25.262916 1140 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d18ed9dc598e4ccaaf8633eaa45a2f51 in term 1.
I20260501 13:57:25.263257 1217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f04a112d3be549d98b4ee77dbefacb7b in term 1.
I20260501 13:57:25.264438 1193 leader_election.cc:304] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d18ed9dc598e4ccaaf8633eaa45a2f51, f04a112d3be549d98b4ee77dbefacb7b; no voters:
I20260501 13:57:25.266276 1117 leader_election.cc:304] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d18ed9dc598e4ccaaf8633eaa45a2f51, f04a112d3be549d98b4ee77dbefacb7b; no voters:
I20260501 13:57:25.270116 1700 leader_election.cc:290] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:25.271394 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7" is_pre_election: true
I20260501 13:57:25.271631 1700 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260501 13:57:25.272001 1700 raft_consensus.cc:493] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:25.272324 1700 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 1 FOLLOWER]: Advancing to term 2
I20260501 13:57:25.276487 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7" is_pre_election: true
I20260501 13:57:25.278811 1701 leader_election.cc:290] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:25.279870 1701 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260501 13:57:25.280798 1701 raft_consensus.cc:493] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 50c0932256e74399a537670bb424a1b7)
I20260501 13:57:25.281864 1701 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 1 FOLLOWER]: Advancing to term 2
I20260501 13:57:25.284750 1700 raft_consensus.cc:515] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:25.287787 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b"
I20260501 13:57:25.288928 1700 leader_election.cc:290] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 election: Requested vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:25.290151 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7"
I20260501 13:57:25.296061 1701 raft_consensus.cc:515] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:25.297395 1140 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d18ed9dc598e4ccaaf8633eaa45a2f51 in current term 2: Already voted for candidate f04a112d3be549d98b4ee77dbefacb7b in this term.
I20260501 13:57:25.299683 1701 leader_election.cc:290] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 2 election: Requested vote from peers d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:25.301234 1217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51"
I20260501 13:57:25.301542 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7"
I20260501 13:57:25.302152 1217 raft_consensus.cc:2393] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f04a112d3be549d98b4ee77dbefacb7b in current term 2: Already voted for candidate d18ed9dc598e4ccaaf8633eaa45a2f51 in this term.
I20260501 13:57:25.303980 1193 leader_election.cc:304] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d18ed9dc598e4ccaaf8633eaa45a2f51; no voters: 50c0932256e74399a537670bb424a1b7, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:25.305054 1700 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260501 13:57:25.304936 1119 leader_election.cc:304] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f04a112d3be549d98b4ee77dbefacb7b; no voters: 50c0932256e74399a537670bb424a1b7, d18ed9dc598e4ccaaf8633eaa45a2f51
I20260501 13:57:25.306432 1701 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260501 13:57:25.308504 1217 raft_consensus.cc:1240] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 2 FOLLOWER]: Rejecting Update request from peer 50c0932256e74399a537670bb424a1b7 for earlier term 1. Current term is 2. Ops: []
I20260501 13:57:25.323149 1705 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260501 13:57:25.333889 1704 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 1 LEADER]: Stepping down as leader of term 1
I20260501 13:57:25.334146 1704 raft_consensus.cc:740] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 50c0932256e74399a537670bb424a1b7, State: Running, Role: LEADER
I20260501 13:57:25.334842 1704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:25.335722 1704 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 1 FOLLOWER]: Advancing to term 2
I20260501 13:57:25.641048 1344 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 13:57:25.825649 1704 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:25.826200 1704 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:25.828665 1704 leader_election.cc:290] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897)
I20260501 13:57:25.831308 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
I20260501 13:57:25.831988 1140 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50c0932256e74399a537670bb424a1b7 in term 2.
I20260501 13:57:25.833149 1217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" is_pre_election: true
I20260501 13:57:25.833693 1217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50c0932256e74399a537670bb424a1b7 in term 2.
I20260501 13:57:25.834002 1707 raft_consensus.cc:493] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:25.834407 1707 raft_consensus.cc:515] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:25.835160 1271 leader_election.cc:304] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50c0932256e74399a537670bb424a1b7, f04a112d3be549d98b4ee77dbefacb7b; no voters:
I20260501 13:57:25.837146 1707 leader_election.cc:290] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:25.837901 1217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" is_pre_election: true
I20260501 13:57:25.838122 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7" is_pre_election: true
I20260501 13:57:25.838441 1217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f04a112d3be549d98b4ee77dbefacb7b in term 2.
I20260501 13:57:25.838781 1297 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f04a112d3be549d98b4ee77dbefacb7b in term 2.
I20260501 13:57:25.839243 1704 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260501 13:57:25.839553 1704 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:25.839872 1704 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 2 FOLLOWER]: Advancing to term 3
I20260501 13:57:25.840220 1117 leader_election.cc:304] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d18ed9dc598e4ccaaf8633eaa45a2f51, f04a112d3be549d98b4ee77dbefacb7b; no voters:
I20260501 13:57:25.842271 1707 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260501 13:57:25.842702 1707 raft_consensus.cc:493] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:25.843082 1707 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 2 FOLLOWER]: Advancing to term 3
I20260501 13:57:25.847870 1704 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:25.850446 1707 raft_consensus.cc:515] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:25.854404 1217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51"
I20260501 13:57:25.855033 1217 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 2 FOLLOWER]: Advancing to term 3
I20260501 13:57:25.857900 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b"
I20260501 13:57:25.858634 1140 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 50c0932256e74399a537670bb424a1b7 in current term 3: Already voted for candidate f04a112d3be549d98b4ee77dbefacb7b in this term.
I20260501 13:57:25.860430 1704 leader_election.cc:290] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 3 election: Requested vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897)
I20260501 13:57:25.861598 1216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51"
I20260501 13:57:25.863415 1270 leader_election.cc:304] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50c0932256e74399a537670bb424a1b7; no voters: d18ed9dc598e4ccaaf8633eaa45a2f51, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:25.864634 1704 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260501 13:57:25.868012 1297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f04a112d3be549d98b4ee77dbefacb7b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "50c0932256e74399a537670bb424a1b7"
I20260501 13:57:25.868804 1297 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f04a112d3be549d98b4ee77dbefacb7b in current term 3: Already voted for candidate 50c0932256e74399a537670bb424a1b7 in this term.
I20260501 13:57:25.870283 1707 leader_election.cc:290] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 3 election: Requested vote from peers d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897), 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815)
I20260501 13:57:25.871948 1217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f04a112d3be549d98b4ee77dbefacb7b in term 3.
I20260501 13:57:25.873131 1117 leader_election.cc:304] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d18ed9dc598e4ccaaf8633eaa45a2f51, f04a112d3be549d98b4ee77dbefacb7b; no voters: 50c0932256e74399a537670bb424a1b7
I20260501 13:57:25.873951 1707 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 3 FOLLOWER]: Leader election won for term 3
I20260501 13:57:25.875118 1707 raft_consensus.cc:697] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [term 3 LEADER]: Becoming Leader. State: Replica: f04a112d3be549d98b4ee77dbefacb7b, State: Running, Role: LEADER
I20260501 13:57:25.876183 1707 consensus_queue.cc:237] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:25.881135 1708 sys_catalog.cc:455] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f04a112d3be549d98b4ee77dbefacb7b. Latest consensus state: current_term: 3 leader_uuid: "f04a112d3be549d98b4ee77dbefacb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:25.882081 1708 sys_catalog.cc:458] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [sys.catalog]: This master's current role is: LEADER
I20260501 13:57:25.883929 1710 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 13:57:25.892143 1710 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 13:57:25.898126 1710 catalog_manager.cc:1269] Loaded cluster ID: 5d0126731b4a4ae2a770b98ff0c83307
I20260501 13:57:25.898447 1710 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 13:57:25.903473 1710 catalog_manager.cc:1514] Loading token signing keys...
I20260501 13:57:25.907603 1710 catalog_manager.cc:6055] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Loaded TSK: 0
I20260501 13:57:25.908985 1710 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260501 13:57:24.778048 1694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:25.966869 1696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:57:25.969271 1695 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1186 milliseconds
I20260501 13:57:25.969417 1685 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260501 13:57:25.970764 1685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:57:25.972920 1685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:57:25.974309 1685 hybrid_clock.cc:648] HybridClock initialized: now 1777643845974263 us; error 47 us; skew 500 ppm
I20260501 13:57:25.975191 1685 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:57:25.981289 1685 webserver.cc:492] Webserver started at http://127.1.16.131:43539/ using document root <none> and password file <none>
I20260501 13:57:25.982255 1685 fs_manager.cc:362] Metadata directory not provided
I20260501 13:57:25.982492 1685 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:57:25.982937 1685 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:57:25.987176 1685 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/data/instance:
uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
format_stamp: "Formatted at 2026-05-01 13:57:25 on dist-test-slave-nqxd"
I20260501 13:57:25.988250 1685 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/instance:
uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
format_stamp: "Formatted at 2026-05-01 13:57:25 on dist-test-slave-nqxd"
I20260501 13:57:25.995428 1685 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260501 13:57:26.001602 1715 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:26.002926 1685 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260501 13:57:26.003280 1685 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal
uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
format_stamp: "Formatted at 2026-05-01 13:57:25 on dist-test-slave-nqxd"
I20260501 13:57:26.003613 1685 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:57:26.053860 1685 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:57:26.055477 1685 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:57:26.055999 1685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:57:26.059037 1685 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 13:57:26.063273 1685 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 13:57:26.063494 1685 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:26.063751 1685 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 13:57:26.063946 1685 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:26.216794 1685 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.131:34859
I20260501 13:57:26.216892 1827 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.131:34859 every 8 connection(s)
I20260501 13:57:26.219421 1685 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/data/info.pb
I20260501 13:57:26.227880 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 1685
I20260501 13:57:26.228417 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/instance
I20260501 13:57:26.262817 1828 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:26.263356 1829 heartbeater.cc:344] Connected to a master server at 127.1.16.189:45897
I20260501 13:57:26.263358 1828 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:26.263391 1830 heartbeater.cc:344] Connected to a master server at 127.1.16.190:45953
I20260501 13:57:26.263731 1829 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:26.263849 1830 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:26.265070 1829 heartbeater.cc:507] Master 127.1.16.189:45897 requested a full tablet report, sending...
I20260501 13:57:26.265187 1828 heartbeater.cc:507] Master 127.1.16.188:38815 requested a full tablet report, sending...
I20260501 13:57:26.265202 1830 heartbeater.cc:507] Master 127.1.16.190:45953 requested a full tablet report, sending...
I20260501 13:57:26.267611 1207 ts_manager.cc:194] Registered new tserver with Master: 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:26.267691 1287 ts_manager.cc:194] Registered new tserver with Master: 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:26.269191 1132 ts_manager.cc:194] Registered new tserver with Master: 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:26.271754 1132 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.16.131:42895
I20260501 13:57:26.281225 1090 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 13:57:26.384800 1217 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [term 3 FOLLOWER]: Refusing update from remote peer f04a112d3be549d98b4ee77dbefacb7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260501 13:57:26.386627 1708 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260501 13:57:26.358982 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_f6554eca1b6f40f39a965ea087497184"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:26.362829 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_ef64b8711a9f43c180ff68dda3adad65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:26.399554 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef64b8711a9f43c180ff68dda3adad65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:26.399577 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6554eca1b6f40f39a965ea087497184 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:26.359253 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_96ecfec755b742b293f7210b103f3a6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:26.358709 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_1b79f7a2bb254f18a8a567f1f9d49b28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:26.404862 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96ecfec755b742b293f7210b103f3a6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:26.405406 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b79f7a2bb254f18a8a567f1f9d49b28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:26.412052 1859 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f04a112d3be549d98b4ee77dbefacb7b. Latest consensus state: current_term: 3 leader_uuid: "f04a112d3be549d98b4ee77dbefacb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:26.412979 1859 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:26.418793 1090 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu with pid 1244
I20260501 13:57:26.424271 1708 sys_catalog.cc:455] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f04a112d3be549d98b4ee77dbefacb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:26.425069 1708 sys_catalog.cc:458] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b [sys.catalog]: This master's current role is: LEADER
I20260501 13:57:26.440891 1859 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f04a112d3be549d98b4ee77dbefacb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:26.441790 1859 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ed9dc598e4ccaaf8633eaa45a2f51 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:26.455210 1535 heartbeater.cc:499] Master 127.1.16.190:45953 was elected leader, sending a full tablet report...
W20260501 13:57:26.458519 1119 connection.cc:570] client connection to 127.1.16.188:38815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:57:26.459004 1119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 13 similar messages]
I20260501 13:57:26.473253 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_f6554eca1b6f40f39a965ea087497184" } new_table_name: "default.Table_b26a5b7250854141889e8708ce698991" modify_external_catalogs: true
I20260501 13:57:26.475767 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_1b79f7a2bb254f18a8a567f1f9d49b28" } new_table_name: "default.Table_4099dfa8c6b84a9fb0fdb9fe6ed3b703" modify_external_catalogs: true
I20260501 13:57:26.477640 1129 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_96ecfec755b742b293f7210b103f3a6b" } modify_external_catalogs: true
I20260501 13:57:26.478308 1129 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_96ecfec755b742b293f7210b103f3a6b" } modify_external_catalogs: true
I20260501 13:57:26.479579 1130 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ef64b8711a9f43c180ff68dda3adad65" } modify_external_catalogs: true
I20260501 13:57:26.480099 1130 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ef64b8711a9f43c180ff68dda3adad65" } modify_external_catalogs: true
W20260501 13:57:26.507074 1872 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 13:57:26.471637 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.1.16.188:38815 (local address 127.1.16.129:34647)
0501 13:57:26.497192 (+ 25555us) negotiation.cc:107] Waiting for socket to connect
0501 13:57:26.497528 (+ 336us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":24913,"thread_start_us":24634,"threads_started":1}
W20260501 13:57:26.507975 1532 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection reset by peer (error 104)
W20260501 13:57:26.509809 1119 consensus_peers.cc:597] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b -> Peer 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815): Couldn't send request to peer 50c0932256e74399a537670bb424a1b7. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:26.516417 1129 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 751927bd15d64ea9bdc3d70550e99bbc
W20260501 13:57:26.517323 1128 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc255bd2d2374f96bec525417a48dbcf has no consensus state
I20260501 13:57:26.524937 1130 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 2e8b420984b745d9b91f54e708118e5c
W20260501 13:57:26.532214 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05378c49253142f28589d42a70235385 has no consensus state
W20260501 13:57:26.554667 1116 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet cc255bd2d2374f96bec525417a48dbcf (table default.Table_4099dfa8c6b84a9fb0fdb9fe6ed3b703 [id=ee3822ae2afd456fbb22e9860e37b240], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet cc255bd2d2374f96bec525417a48dbcf consensus state has no leader
I20260501 13:57:26.575139 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 05378c49253142f28589d42a70235385 (DEFAULT_TABLE table=default.Table_b26a5b7250854141889e8708ce698991 [id=870dc36eb99c4cba874be3125bda1ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:26.577281 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05378c49253142f28589d42a70235385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:26.576537 1465 tablet_service.cc:1511] Processing CreateTablet for tablet cc255bd2d2374f96bec525417a48dbcf (DEFAULT_TABLE table=default.Table_4099dfa8c6b84a9fb0fdb9fe6ed3b703 [id=ee3822ae2afd456fbb22e9860e37b240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:26.578552 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc255bd2d2374f96bec525417a48dbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:26.578416 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 05378c49253142f28589d42a70235385 (DEFAULT_TABLE table=default.Table_b26a5b7250854141889e8708ce698991 [id=870dc36eb99c4cba874be3125bda1ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:26.579591 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05378c49253142f28589d42a70235385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:26.573285 1610 tablet_service.cc:1511] Processing CreateTablet for tablet cc255bd2d2374f96bec525417a48dbcf (DEFAULT_TABLE table=default.Table_4099dfa8c6b84a9fb0fdb9fe6ed3b703 [id=ee3822ae2afd456fbb22e9860e37b240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:26.583927 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc255bd2d2374f96bec525417a48dbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:26.604452 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 05378c49253142f28589d42a70235385 (DEFAULT_TABLE table=default.Table_b26a5b7250854141889e8708ce698991 [id=870dc36eb99c4cba874be3125bda1ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:26.606962 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05378c49253142f28589d42a70235385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:26.601140 1763 tablet_service.cc:1511] Processing CreateTablet for tablet cc255bd2d2374f96bec525417a48dbcf (DEFAULT_TABLE table=default.Table_4099dfa8c6b84a9fb0fdb9fe6ed3b703 [id=ee3822ae2afd456fbb22e9860e37b240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:26.611914 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc255bd2d2374f96bec525417a48dbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:26.615788 1881 tablet_bootstrap.cc:492] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:26.632473 1882 tablet_bootstrap.cc:492] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:26.634222 1881 tablet_bootstrap.cc:654] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:26.636897 1881 log.cc:826] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Log is configured to *not* fsync() on all Append() calls
I20260501 13:57:26.641127 1882 tablet_bootstrap.cc:654] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:26.642086 1883 tablet_bootstrap.cc:492] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:26.644547 1882 log.cc:826] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Log is configured to *not* fsync() on all Append() calls
I20260501 13:57:26.646777 1881 tablet_bootstrap.cc:492] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:26.647318 1881 ts_tablet_manager.cc:1403] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.015s
I20260501 13:57:26.650141 1883 tablet_bootstrap.cc:654] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:26.652540 1883 log.cc:826] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Log is configured to *not* fsync() on all Append() calls
I20260501 13:57:26.659869 1883 tablet_bootstrap.cc:492] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:26.660645 1883 ts_tablet_manager.cc:1403] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.003s
I20260501 13:57:26.661967 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7373e38f4327497385cc7454033934ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:26.663874 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7373e38f4327497385cc7454033934ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:26.667994 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_e6554560868d4edf9805cad585feb816"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:26.669731 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6554560868d4edf9805cad585feb816 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:26.675275 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_36d3889bfbba4903b69bc895e418ffb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:26.676051 1881 raft_consensus.cc:359] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:26.676964 1881 raft_consensus.cc:385] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260501 13:57:26.677089 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36d3889bfbba4903b69bc895e418ffb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:26.677266 1881 raft_consensus.cc:740] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:26.678251 1881 consensus_queue.cc:260] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:26.678973 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_17e52f797589413ab3981e21e3a96045"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:26.681295 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17e52f797589413ab3981e21e3a96045 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:26.691104 1883 raft_consensus.cc:359] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:26.692318 1883 raft_consensus.cc:385] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:26.692830 1883 raft_consensus.cc:740] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:26.694089 1883 consensus_queue.cc:260] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:26.720947 1131 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4099dfa8c6b84a9fb0fdb9fe6ed3b703" } modify_external_catalogs: true
I20260501 13:57:26.721593 1131 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4099dfa8c6b84a9fb0fdb9fe6ed3b703" } modify_external_catalogs: true
I20260501 13:57:26.727531 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_b26a5b7250854141889e8708ce698991" } new_table_name: "default.Table_07d6a87bcbaa42238a5061bbc0c4bf32" modify_external_catalogs: true
I20260501 13:57:26.775323 1830 heartbeater.cc:499] Master 127.1.16.190:45953 was elected leader, sending a full tablet report...
I20260501 13:57:26.780984 1882 tablet_bootstrap.cc:492] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:26.781742 1882 ts_tablet_manager.cc:1403] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.151s user 0.016s sys 0.008s
W20260501 13:57:26.798385 1828 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:26.805887 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_e6554560868d4edf9805cad585feb816" } new_table_name: "default.Table_26aa93a946694ca2b789c7eb59dc60c2" modify_external_catalogs: true
I20260501 13:57:26.824522 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_17e52f797589413ab3981e21e3a96045" } modify_external_catalogs: true
I20260501 13:57:26.825156 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_17e52f797589413ab3981e21e3a96045" } modify_external_catalogs: true
I20260501 13:57:26.878911 1881 ts_tablet_manager.cc:1434] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.231s user 0.090s sys 0.110s
I20260501 13:57:26.879597 1883 ts_tablet_manager.cc:1434] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.219s user 0.098s sys 0.112s
I20260501 13:57:26.882057 1883 tablet_bootstrap.cc:492] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:26.885260 1881 tablet_bootstrap.cc:492] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:26.900986 1881 tablet_bootstrap.cc:654] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:26.907518 1883 tablet_bootstrap.cc:654] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:26.909150 1678 heartbeater.cc:499] Master 127.1.16.190:45953 was elected leader, sending a full tablet report...
I20260501 13:57:26.914124 1888 raft_consensus.cc:493] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 13:57:26.915465 1125 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7704a273008b47da9b158e280c91f06d has no consensus state
I20260501 13:57:26.915290 1888 raft_consensus.cc:515] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:26.933108 1888 leader_election.cc:290] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:26.935150 1129 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_b26a5b7250854141889e8708ce698991 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 1 got 0
I20260501 13:57:26.937328 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 495f5f95791b4817b50a60efc50fb30c (DEFAULT_TABLE table=default.Table_17e52f797589413ab3981e21e3a96045 [id=322103fbe9bb423999ba4858a447cbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:26.938570 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 495f5f95791b4817b50a60efc50fb30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:26.943020 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05378c49253142f28589d42a70235385" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:26.943265 1128 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_b26a5b7250854141889e8708ce698991 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 1 got 0
I20260501 13:57:26.943956 1485 raft_consensus.cc:2468] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 0.
I20260501 13:57:26.945218 1132 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_b26a5b7250854141889e8708ce698991 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 1 got 0
I20260501 13:57:26.946815 1717 leader_election.cc:304] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters:
W20260501 13:57:26.948151 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05378c49253142f28589d42a70235385 consensus state has no leader
I20260501 13:57:26.949203 1131 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet cc255bd2d2374f96bec525417a48dbcf
I20260501 13:57:26.949501 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 495f5f95791b4817b50a60efc50fb30c (DEFAULT_TABLE table=default.Table_17e52f797589413ab3981e21e3a96045 [id=322103fbe9bb423999ba4858a447cbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:26.950771 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 495f5f95791b4817b50a60efc50fb30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:26.952585 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet cc255bd2d2374f96bec525417a48dbcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:26 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:26.957692 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_8c838ea23108487f967aa19f774f9453"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:26.959405 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c838ea23108487f967aa19f774f9453 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:26.966516 1888 raft_consensus.cc:2804] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:26.966917 1888 raft_consensus.cc:493] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:26.967252 1888 raft_consensus.cc:3060] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:26.971573 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet cc255bd2d2374f96bec525417a48dbcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:26 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:26.990329 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 495f5f95791b4817b50a60efc50fb30c
I20260501 13:57:26.991010 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_ba63dcf55ddc4a66905b08564e2b1ad2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:26.993072 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba63dcf55ddc4a66905b08564e2b1ad2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:26.993304 1608 tablet_service.cc:1558] Processing DeleteTablet for tablet 495f5f95791b4817b50a60efc50fb30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:26 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:26.996649 1835 tablet.cc:2404] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:57:26.999008 1463 tablet_service.cc:1558] Processing DeleteTablet for tablet 495f5f95791b4817b50a60efc50fb30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:26 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:27.004534 1881 tablet_bootstrap.cc:492] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:27.006078 1881 ts_tablet_manager.cc:1403] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.121s user 0.014s sys 0.045s
I20260501 13:57:27.009697 1881 raft_consensus.cc:359] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.010363 1881 raft_consensus.cc:385] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.010687 1881 raft_consensus.cc:740] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.011372 1881 consensus_queue.cc:260] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.012372 1882 raft_consensus.cc:359] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.013237 1882 raft_consensus.cc:385] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.013521 1882 raft_consensus.cc:740] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.016958 1881 ts_tablet_manager.cc:1434] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260501 13:57:27.018429 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 495f5f95791b4817b50a60efc50fb30c (DEFAULT_TABLE table=default.Table_17e52f797589413ab3981e21e3a96045 [id=322103fbe9bb423999ba4858a447cbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.019569 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 495f5f95791b4817b50a60efc50fb30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.022799 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet cc255bd2d2374f96bec525417a48dbcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:26 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:27.024794 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 495f5f95791b4817b50a60efc50fb30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:26 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:27.026106 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05378c49253142f28589d42a70235385" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:27.034089 1888 raft_consensus.cc:515] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.036252 1888 leader_election.cc:290] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:27.037050 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05378c49253142f28589d42a70235385" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:27.037590 1485 raft_consensus.cc:3060] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:27.038160 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05378c49253142f28589d42a70235385" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
W20260501 13:57:27.041054 1675 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38815 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111)
I20260501 13:57:27.014390 1882 consensus_queue.cc:260] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.043946 1630 raft_consensus.cc:2468] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 0.
I20260501 13:57:27.044209 1485 raft_consensus.cc:2468] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 1.
I20260501 13:57:27.046751 1883 tablet_bootstrap.cc:492] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:27.047173 1883 ts_tablet_manager.cc:1403] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.165s user 0.025s sys 0.014s
I20260501 13:57:27.046564 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_764f689a7c8b4b1e9791efa01e15286f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:27.049765 1883 raft_consensus.cc:359] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.050385 1883 raft_consensus.cc:385] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.050688 1883 raft_consensus.cc:740] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.051169 1717 leader_election.cc:304] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:27.052465 1888 raft_consensus.cc:2804] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election won for term 1
W20260501 13:57:27.058406 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet cc255bd2d2374f96bec525417a48dbcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc255bd2d2374f96bec525417a48dbcf already in progress: creating tablet
I20260501 13:57:27.051385 1883 consensus_queue.cc:260] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
W20260501 13:57:27.060473 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 495f5f95791b4817b50a60efc50fb30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 495f5f95791b4817b50a60efc50fb30c already in progress: creating tablet
I20260501 13:57:27.048084 1898 tablet_replica.cc:333] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:27.062134 1898 raft_consensus.cc:2243] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:27.062824 1898 raft_consensus.cc:2272] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:27.065660 1898 ts_tablet_manager.cc:1916] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:27.076272 1888 raft_consensus.cc:697] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Running, Role: LEADER
I20260501 13:57:27.077255 1888 consensus_queue.cc:237] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:27.078192 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_764f689a7c8b4b1e9791efa01e15286f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:27.082885 1883 ts_tablet_manager.cc:1434] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.035s user 0.007s sys 0.003s
I20260501 13:57:27.088846 1899 tablet_replica.cc:333] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:27.089689 1899 raft_consensus.cc:2243] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:27.090447 1883 tablet_bootstrap.cc:492] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:27.092579 1899 raft_consensus.cc:2272] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:27.101353 1883 tablet_bootstrap.cc:654] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:27.103186 1899 ts_tablet_manager.cc:1916] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:27.114609 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_e71908fa21b440cea454e850fa601a0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:27.116384 1882 ts_tablet_manager.cc:1434] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.327s user 0.050s sys 0.019s
W20260501 13:57:27.116290 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e71908fa21b440cea454e850fa601a0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:27.117942 1132 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_07d6a87bcbaa42238a5061bbc0c4bf32 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 2 got 0
I20260501 13:57:27.119427 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_07d6a87bcbaa42238a5061bbc0c4bf32 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 2 got 0
I20260501 13:57:27.125252 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_36d3889bfbba4903b69bc895e418ffb0" } new_table_name: "default.Table_c79a5c7717954055a8ff413892d23933" modify_external_catalogs: true
W20260501 13:57:27.127243 1537 tablet.cc:2404] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:57:27.129002 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_07d6a87bcbaa42238a5061bbc0c4bf32 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 2 got 0
I20260501 13:57:27.133399 1132 catalog_manager.cc:5671] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: HEALTHY } } }
I20260501 13:57:27.135334 1132 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_07d6a87bcbaa42238a5061bbc0c4bf32 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 2 got 0
I20260501 13:57:27.140340 1882 tablet_bootstrap.cc:492] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:27.146346 1882 tablet_bootstrap.cc:654] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:27.150060 1126 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3ca39bdb92b44986a6ffabbe2d33434c has no consensus state
I20260501 13:57:27.150486 1883 tablet_bootstrap.cc:492] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:27.150966 1883 ts_tablet_manager.cc:1403] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.061s user 0.031s sys 0.009s
I20260501 13:57:27.153611 1883 raft_consensus.cc:359] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.154341 1883 raft_consensus.cc:385] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.154646 1883 raft_consensus.cc:740] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.155468 1883 consensus_queue.cc:260] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.159451 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_7373e38f4327497385cc7454033934ff" } new_table_name: "default.Table_c9015fd7203240529f60abbaf148a94a" modify_external_catalogs: true
I20260501 13:57:27.163738 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_26aa93a946694ca2b789c7eb59dc60c2" } modify_external_catalogs: true
I20260501 13:57:27.164250 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_26aa93a946694ca2b789c7eb59dc60c2" } modify_external_catalogs: true
I20260501 13:57:27.164892 1883 ts_tablet_manager.cc:1434] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260501 13:57:27.165663 1899 ts_tablet_manager.cc:1929] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:27.166062 1899 log.cc:1199] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/cc255bd2d2374f96bec525417a48dbcf
I20260501 13:57:27.166931 1899 ts_tablet_manager.cc:1950] T cc255bd2d2374f96bec525417a48dbcf P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:27.187529 1610 tablet_service.cc:1511] Processing CreateTablet for tablet bbd44eebeb0347b9ac3ecfbdfc209d14 (DEFAULT_TABLE table=default.Table_7373e38f4327497385cc7454033934ff [id=dcc16a2384604fe280c4b99229047c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.188793 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd44eebeb0347b9ac3ecfbdfc209d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.191974 1763 tablet_service.cc:1511] Processing CreateTablet for tablet bbd44eebeb0347b9ac3ecfbdfc209d14 (DEFAULT_TABLE table=default.Table_7373e38f4327497385cc7454033934ff [id=dcc16a2384604fe280c4b99229047c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.194334 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd44eebeb0347b9ac3ecfbdfc209d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.195273 1881 tablet_bootstrap.cc:492] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:27.202730 1881 tablet_bootstrap.cc:654] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:27.212261 1465 tablet_service.cc:1511] Processing CreateTablet for tablet bbd44eebeb0347b9ac3ecfbdfc209d14 (DEFAULT_TABLE table=default.Table_7373e38f4327497385cc7454033934ff [id=dcc16a2384604fe280c4b99229047c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.213528 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd44eebeb0347b9ac3ecfbdfc209d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.210791 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_fe2abe8bde8846438f0be465b6c5cc2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:27.220499 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bbd44eebeb0347b9ac3ecfbdfc209d14 consensus state has no leader
I20260501 13:57:27.227460 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 3ca39bdb92b44986a6ffabbe2d33434c (DEFAULT_TABLE table=default.Table_c79a5c7717954055a8ff413892d23933 [id=9c8ebe4a498847a5a88c42e16f49d510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.228624 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca39bdb92b44986a6ffabbe2d33434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.232275 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet cc255bd2d2374f96bec525417a48dbcf (table default.Table_4099dfa8c6b84a9fb0fdb9fe6ed3b703 [id=ee3822ae2afd456fbb22e9860e37b240]) successfully deleted
W20260501 13:57:27.235620 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe2abe8bde8846438f0be465b6c5cc2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:27.237360 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 7704a273008b47da9b158e280c91f06d
I20260501 13:57:27.240164 1608 tablet_service.cc:1558] Processing DeleteTablet for tablet 7704a273008b47da9b158e280c91f06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:27.240964 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet 7704a273008b47da9b158e280c91f06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:27.243753 1463 tablet_service.cc:1558] Processing DeleteTablet for tablet 7704a273008b47da9b158e280c91f06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:27.243989 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_cbd8e10c95214c048b626e6bb5e171e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:27.245904 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbd8e10c95214c048b626e6bb5e171e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:27.249668 1119 catalog_manager.cc:4969] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 7704a273008b47da9b158e280c91f06d because the tablet was not found. No further retry: Not found: Tablet not found: 7704a273008b47da9b158e280c91f06d
I20260501 13:57:27.253000 1899 tablet_replica.cc:333] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:27.253784 1899 raft_consensus.cc:2243] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:27.253777 1129 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_8c838ea23108487f967aa19f774f9453" } modify_external_catalogs: true
I20260501 13:57:27.254276 1899 raft_consensus.cc:2272] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:27.254451 1129 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_8c838ea23108487f967aa19f774f9453" } modify_external_catalogs: true
I20260501 13:57:27.269358 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 3ca39bdb92b44986a6ffabbe2d33434c (DEFAULT_TABLE table=default.Table_c79a5c7717954055a8ff413892d23933 [id=9c8ebe4a498847a5a88c42e16f49d510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.270591 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca39bdb92b44986a6ffabbe2d33434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.273749 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 3ca39bdb92b44986a6ffabbe2d33434c (DEFAULT_TABLE table=default.Table_c79a5c7717954055a8ff413892d23933 [id=9c8ebe4a498847a5a88c42e16f49d510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.274803 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca39bdb92b44986a6ffabbe2d33434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.293138 1899 ts_tablet_manager.cc:1916] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:27.293964 1608 tablet_service.cc:1511] Processing CreateTablet for tablet 7704a273008b47da9b158e280c91f06d (DEFAULT_TABLE table=default.Table_26aa93a946694ca2b789c7eb59dc60c2 [id=6b6cfcedadc0495d9936a8624b7f62f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.295379 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7704a273008b47da9b158e280c91f06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.303898 1630 raft_consensus.cc:3060] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:27.310690 1630 raft_consensus.cc:1275] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Refusing update from remote peer 0ed1f9a545474e76a661c32bf11a7fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:57:27.300158 1883 tablet_bootstrap.cc:492] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:27.312513 1888 consensus_queue.cc:1048] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:27.322340 1129 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 190bfe5aece84552acd8d9198e3e40a8
I20260501 13:57:27.333395 1883 tablet_bootstrap.cc:654] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:27.338994 1463 tablet_service.cc:1511] Processing CreateTablet for tablet 7704a273008b47da9b158e280c91f06d (DEFAULT_TABLE table=default.Table_26aa93a946694ca2b789c7eb59dc60c2 [id=6b6cfcedadc0495d9936a8624b7f62f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.343073 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 7704a273008b47da9b158e280c91f06d (DEFAULT_TABLE table=default.Table_26aa93a946694ca2b789c7eb59dc60c2 [id=6b6cfcedadc0495d9936a8624b7f62f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.344288 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7704a273008b47da9b158e280c91f06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.344530 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7704a273008b47da9b158e280c91f06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.347685 1485 raft_consensus.cc:1275] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Refusing update from remote peer 0ed1f9a545474e76a661c32bf11a7fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:57:27.349603 1902 consensus_queue.cc:1048] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:57:27.353596 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ba63dcf55ddc4a66905b08564e2b1ad2" } new_table_name: "default.Table_b6ea52503c0b4632a20c8e571d0d4ed8" modify_external_catalogs: true
I20260501 13:57:27.354136 1899 ts_tablet_manager.cc:1929] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:27.357285 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_b34565681ce74d37bbb0ebc07baff2e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:27.358982 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b34565681ce74d37bbb0ebc07baff2e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:27.361920 1881 tablet_bootstrap.cc:492] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:27.362323 1881 ts_tablet_manager.cc:1403] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.167s user 0.008s sys 0.006s
I20260501 13:57:27.364862 1881 raft_consensus.cc:359] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.367555 1899 log.cc:1199] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/495f5f95791b4817b50a60efc50fb30c
I20260501 13:57:27.368263 1899 ts_tablet_manager.cc:1950] T 495f5f95791b4817b50a60efc50fb30c P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:27.374877 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 495f5f95791b4817b50a60efc50fb30c (table default.Table_17e52f797589413ab3981e21e3a96045 [id=322103fbe9bb423999ba4858a447cbb4]) successfully deleted
I20260501 13:57:27.375186 1881 raft_consensus.cc:385] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.375478 1881 raft_consensus.cc:740] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.376091 1881 consensus_queue.cc:260] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.386962 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4e1d3a6887b14eceae55db865debc857"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:27.388746 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e1d3a6887b14eceae55db865debc857 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:27.391609 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 7704a273008b47da9b158e280c91f06d because the tablet was not found. No further retry: Not found: Tablet not found: 7704a273008b47da9b158e280c91f06d
I20260501 13:57:27.393936 1883 tablet_bootstrap.cc:492] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:27.395100 1883 ts_tablet_manager.cc:1403] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.095s user 0.015s sys 0.010s
I20260501 13:57:27.398253 1883 raft_consensus.cc:359] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.399024 1883 raft_consensus.cc:385] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.399348 1883 raft_consensus.cc:740] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.400098 1883 consensus_queue.cc:260] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.412024 1883 ts_tablet_manager.cc:1434] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.017s user 0.001s sys 0.005s
I20260501 13:57:27.413553 1131 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 3ca39bdb92b44986a6ffabbe2d33434c (table default.Table_c79a5c7717954055a8ff413892d23933 [id=9c8ebe4a498847a5a88c42e16f49d510]). Expected version 1 got 0
I20260501 13:57:27.425295 1881 ts_tablet_manager.cc:1434] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.063s user 0.004s sys 0.001s
I20260501 13:57:27.426226 1881 tablet_bootstrap.cc:492] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:27.426548 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_07d6a87bcbaa42238a5061bbc0c4bf32" } new_table_name: "default.Table_d746091594f546cc8a653e0653623394" modify_external_catalogs: true
I20260501 13:57:27.429505 1898 ts_tablet_manager.cc:1929] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:27.429927 1898 log.cc:1199] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/cc255bd2d2374f96bec525417a48dbcf
I20260501 13:57:27.431041 1898 ts_tablet_manager.cc:1950] T cc255bd2d2374f96bec525417a48dbcf P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:27.432261 1898 tablet_replica.cc:333] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:27.432901 1898 raft_consensus.cc:2243] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:27.433312 1898 raft_consensus.cc:2272] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:27.433465 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet cc255bd2d2374f96bec525417a48dbcf (table default.Table_4099dfa8c6b84a9fb0fdb9fe6ed3b703 [id=ee3822ae2afd456fbb22e9860e37b240]) successfully deleted
I20260501 13:57:27.435618 1898 ts_tablet_manager.cc:1916] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:27.437875 1881 tablet_bootstrap.cc:654] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:27.447217 1883 tablet_bootstrap.cc:492] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:27.456820 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 3ca39bdb92b44986a6ffabbe2d33434c (table default.Table_c79a5c7717954055a8ff413892d23933 [id=9c8ebe4a498847a5a88c42e16f49d510]). Expected version 1 got 0
I20260501 13:57:27.462345 1881 tablet_bootstrap.cc:492] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:27.462836 1907 mvcc.cc:204] Tried to move back new op lower bound from 7281229198531715072 to 7281229197637672960. Current Snapshot: MvccSnapshot[applied={T|T < 7281229198531715072}]
I20260501 13:57:27.463480 1881 ts_tablet_manager.cc:1403] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.000s
I20260501 13:57:27.471695 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_fe2abe8bde8846438f0be465b6c5cc2d" } modify_external_catalogs: true
I20260501 13:57:27.473194 1883 tablet_bootstrap.cc:654] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:27.474371 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_fe2abe8bde8846438f0be465b6c5cc2d" } modify_external_catalogs: true
I20260501 13:57:27.466010 1881 raft_consensus.cc:359] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.475208 1881 raft_consensus.cc:385] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.475886 1881 raft_consensus.cc:740] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.477015 1881 consensus_queue.cc:260] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.481339 1881 ts_tablet_manager.cc:1434] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.018s user 0.003s sys 0.002s
I20260501 13:57:27.483224 1132 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_07d6a87bcbaa42238a5061bbc0c4bf32 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 2 got 0
I20260501 13:57:27.487118 1881 tablet_bootstrap.cc:492] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:27.491645 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_b9fc1d60ce8f47738143b74f0a59edf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:27.491580 1914 tablet.cc:1724] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:27.492259 1883 tablet_bootstrap.cc:492] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:27.492697 1883 ts_tablet_manager.cc:1403] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.000s
W20260501 13:57:27.493389 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9fc1d60ce8f47738143b74f0a59edf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:27.495270 1883 raft_consensus.cc:359] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.495923 1883 raft_consensus.cc:385] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.495515 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_76959fdf8538441aa44f9e120288cdab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:27.499495 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76959fdf8538441aa44f9e120288cdab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:27.502066 1883 raft_consensus.cc:740] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.502132 1898 ts_tablet_manager.cc:1929] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:27.502516 1898 log.cc:1199] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/495f5f95791b4817b50a60efc50fb30c
I20260501 13:57:27.502784 1883 consensus_queue.cc:260] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.503526 1881 tablet_bootstrap.cc:654] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:27.504828 1883 ts_tablet_manager.cc:1434] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260501 13:57:27.503297 1898 ts_tablet_manager.cc:1950] T 495f5f95791b4817b50a60efc50fb30c P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
W20260501 13:57:27.509886 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 921e5c3eaaa44c62abbc90f560049614 has no consensus state
I20260501 13:57:27.512773 1901 raft_consensus.cc:493] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:27.513229 1901 raft_consensus.cc:515] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.515861 1901 leader_election.cc:290] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:27.517719 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca39bdb92b44986a6ffabbe2d33434c" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:27.519176 1717 leader_election.cc:343] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:27.521059 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 495f5f95791b4817b50a60efc50fb30c (table default.Table_17e52f797589413ab3981e21e3a96045 [id=322103fbe9bb423999ba4858a447cbb4]) successfully deleted
I20260501 13:57:27.521056 1913 tablet.cc:1724] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260501 13:57:27.527976 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 7704a273008b47da9b158e280c91f06d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7704a273008b47da9b158e280c91f06d already in progress: creating tablet
W20260501 13:57:27.523442 1684 tablet.cc:2404] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:57:27.532218 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca39bdb92b44986a6ffabbe2d33434c" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:27.535612 1881 tablet_bootstrap.cc:492] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:27.536077 1881 ts_tablet_manager.cc:1403] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.002s
W20260501 13:57:27.538502 1719 leader_election.cc:343] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:27.538934 1719 leader_election.cc:304] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:27.540714 1882 tablet_bootstrap.cc:492] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:27.541270 1882 ts_tablet_manager.cc:1403] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.401s user 0.056s sys 0.072s
I20260501 13:57:27.541244 1881 raft_consensus.cc:359] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.541857 1881 raft_consensus.cc:385] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.542318 1881 raft_consensus.cc:740] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.543586 1881 consensus_queue.cc:260] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.544315 1902 raft_consensus.cc:2749] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:27.545908 1882 raft_consensus.cc:359] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.546551 1882 raft_consensus.cc:385] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.546819 1882 raft_consensus.cc:740] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.547367 1882 consensus_queue.cc:260] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.549479 1882 ts_tablet_manager.cc:1434] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260501 13:57:27.550269 1882 tablet_bootstrap.cc:492] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:27.553303 1129 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c9015fd7203240529f60abbaf148a94a" } modify_external_catalogs: true
I20260501 13:57:27.553843 1129 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c9015fd7203240529f60abbaf148a94a" } modify_external_catalogs: true
I20260501 13:57:27.555553 1881 ts_tablet_manager.cc:1434] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20260501 13:57:27.556528 1881 tablet_bootstrap.cc:492] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:27.559640 1883 tablet_bootstrap.cc:492] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:27.565748 1883 tablet_bootstrap.cc:654] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:27.565791 1882 tablet_bootstrap.cc:654] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:27.576670 1116 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_b6ea52503c0b4632a20c8e571d0d4ed8 [id=7ed508c1fff74966aa6f804d059b9c47], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 921e5c3eaaa44c62abbc90f560049614 has no consensus state
I20260501 13:57:27.603637 1881 tablet_bootstrap.cc:654] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:27.616897 1909 mvcc.cc:204] Tried to move back new op lower bound from 7281229198531715072 to 7281229197637672960. Current Snapshot: MvccSnapshot[applied={T|T < 7281229198531715072}]
I20260501 13:57:27.617391 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 871448e01d554cc999ffa3099afeaa83 (DEFAULT_TABLE table=default.Table_cbd8e10c95214c048b626e6bb5e171e8 [id=315879bd9f894b148f513e0b72d1d435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.617909 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 871448e01d554cc999ffa3099afeaa83 (DEFAULT_TABLE table=default.Table_cbd8e10c95214c048b626e6bb5e171e8 [id=315879bd9f894b148f513e0b72d1d435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.618455 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871448e01d554cc999ffa3099afeaa83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.621238 1883 tablet_bootstrap.cc:492] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:27.622041 1883 ts_tablet_manager.cc:1403] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.063s user 0.018s sys 0.005s
I20260501 13:57:27.625824 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 14fba31b7fdd42818f727fd2a745908d (DEFAULT_TABLE table=default.Table_764f689a7c8b4b1e9791efa01e15286f [id=e4b3b630ee3a409f8efa7d0612d3a2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.628960 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4b0e296f9b2046e7a9a250d3211a06b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:27.630677 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b0e296f9b2046e7a9a250d3211a06b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:27.632933 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871448e01d554cc999ffa3099afeaa83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.631910 1883 raft_consensus.cc:359] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.634047 1883 raft_consensus.cc:385] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.634619 1883 raft_consensus.cc:740] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.635957 1883 consensus_queue.cc:260] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.637369 1900 raft_consensus.cc:493] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:27.637857 1900 raft_consensus.cc:515] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.638432 1131 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_d746091594f546cc8a653e0653623394 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 3 got 2
I20260501 13:57:27.642102 1463 tablet_service.cc:1511] Processing CreateTablet for tablet 14fba31b7fdd42818f727fd2a745908d (DEFAULT_TABLE table=default.Table_764f689a7c8b4b1e9791efa01e15286f [id=e4b3b630ee3a409f8efa7d0612d3a2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.643044 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fba31b7fdd42818f727fd2a745908d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.643230 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fba31b7fdd42818f727fd2a745908d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.645037 1129 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet bbd44eebeb0347b9ac3ecfbdfc209d14
I20260501 13:57:27.646198 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 14fba31b7fdd42818f727fd2a745908d (DEFAULT_TABLE table=default.Table_764f689a7c8b4b1e9791efa01e15286f [id=e4b3b630ee3a409f8efa7d0612d3a2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.647261 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fba31b7fdd42818f727fd2a745908d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.647445 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet bbd44eebeb0347b9ac3ecfbdfc209d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:27.648077 1898 tablet_replica.cc:333] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:27.648454 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet bbd44eebeb0347b9ac3ecfbdfc209d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:27.648782 1898 raft_consensus.cc:2243] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:27.649212 1898 raft_consensus.cc:2272] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:27.651325 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet bbd44eebeb0347b9ac3ecfbdfc209d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:27.652980 1899 tablet_replica.cc:333] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:27.653666 1899 raft_consensus.cc:2243] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:27.655453 1899 raft_consensus.cc:2272] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:27.660750 1131 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_d746091594f546cc8a653e0653623394 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 3 got 2
I20260501 13:57:27.670719 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 921e5c3eaaa44c62abbc90f560049614 (DEFAULT_TABLE table=default.Table_b6ea52503c0b4632a20c8e571d0d4ed8 [id=7ed508c1fff74966aa6f804d059b9c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.670202 1920 tablet.cc:1724] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:27.675132 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet 7704a273008b47da9b158e280c91f06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:27.674798 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921e5c3eaaa44c62abbc90f560049614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.677997 1610 tablet_service.cc:1511] Processing CreateTablet for tablet 921e5c3eaaa44c62abbc90f560049614 (DEFAULT_TABLE table=default.Table_b6ea52503c0b4632a20c8e571d0d4ed8 [id=7ed508c1fff74966aa6f804d059b9c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.678961 1898 ts_tablet_manager.cc:1916] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:27.679083 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921e5c3eaaa44c62abbc90f560049614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.679287 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 14b221e0725647df9789c2137ababefe
I20260501 13:57:27.682189 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 14b221e0725647df9789c2137ababefe (DEFAULT_TABLE table=default.Table_fe2abe8bde8846438f0be465b6c5cc2d [id=2d510b353dcc40938a2a8c3e34c99b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.683315 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b221e0725647df9789c2137ababefe. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:27.683404 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet bbd44eebeb0347b9ac3ecfbdfc209d14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbd44eebeb0347b9ac3ecfbdfc209d14 already in progress: creating tablet
I20260501 13:57:27.689046 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_de009c8498fe43d8a34a119924faa8d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:27.690368 1460 tablet_service.cc:1558] Processing DeleteTablet for tablet 14b221e0725647df9789c2137ababefe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:27.693902 1899 ts_tablet_manager.cc:1916] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:27.694444 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 921e5c3eaaa44c62abbc90f560049614 (DEFAULT_TABLE table=default.Table_b6ea52503c0b4632a20c8e571d0d4ed8 [id=7ed508c1fff74966aa6f804d059b9c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.695530 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921e5c3eaaa44c62abbc90f560049614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.698879 1760 tablet_service.cc:1558] Processing DeleteTablet for tablet 7704a273008b47da9b158e280c91f06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:27.700660 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 14b221e0725647df9789c2137ababefe (DEFAULT_TABLE table=default.Table_fe2abe8bde8846438f0be465b6c5cc2d [id=2d510b353dcc40938a2a8c3e34c99b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.702414 1461 tablet_service.cc:1511] Processing CreateTablet for tablet d74cf3b82ecf4379a81a1e2ec39213c9 (DEFAULT_TABLE table=default.Table_e71908fa21b440cea454e850fa601a0a [id=9bf3fd03858a42a2ba91370283546ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.703883 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d74cf3b82ecf4379a81a1e2ec39213c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.704102 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 14b221e0725647df9789c2137ababefe (DEFAULT_TABLE table=default.Table_fe2abe8bde8846438f0be465b6c5cc2d [id=2d510b353dcc40938a2a8c3e34c99b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.706176 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b221e0725647df9789c2137ababefe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.701911 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_d746091594f546cc8a653e0653623394 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 3 got 0
I20260501 13:57:27.716881 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_cbd8e10c95214c048b626e6bb5e171e8" } new_table_name: "default.Table_f25950402847469599f123ae30ac49f2" modify_external_catalogs: true
W20260501 13:57:27.714020 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de009c8498fe43d8a34a119924faa8d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:27.719570 1606 tablet_service.cc:1511] Processing CreateTablet for tablet d74cf3b82ecf4379a81a1e2ec39213c9 (DEFAULT_TABLE table=default.Table_e71908fa21b440cea454e850fa601a0a [id=9bf3fd03858a42a2ba91370283546ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.720700 1608 tablet_service.cc:1511] Processing CreateTablet for tablet 871448e01d554cc999ffa3099afeaa83 (DEFAULT_TABLE table=default.Table_cbd8e10c95214c048b626e6bb5e171e8 [id=315879bd9f894b148f513e0b72d1d435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.703850 1900 leader_election.cc:290] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:27.730772 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet 14b221e0725647df9789c2137ababefe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:27.739498 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 05378c49253142f28589d42a70235385 (table default.Table_d746091594f546cc8a653e0653623394 [id=870dc36eb99c4cba874be3125bda1ca9]). Expected version 3 got 2
I20260501 13:57:27.741331 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d74cf3b82ecf4379a81a1e2ec39213c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:27.747727 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 871448e01d554cc999ffa3099afeaa83 consensus state has no leader
I20260501 13:57:27.752516 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4b0e296f9b2046e7a9a250d3211a06b6" } new_table_name: "default.Table_7d92177d90f641498ccbdef2223fa7e4" modify_external_catalogs: true
I20260501 13:57:27.757670 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871448e01d554cc999ffa3099afeaa83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.761807 1460 tablet_service.cc:1511] Processing CreateTablet for tablet d2c2873572de4889a16fda8fe31ad594 (DEFAULT_TABLE table=default.Table_4b0e296f9b2046e7a9a250d3211a06b6 [id=b8901f06222746b8bf418391866b92cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.763769 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c2873572de4889a16fda8fe31ad594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.769146 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b221e0725647df9789c2137ababefe. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:27.770766 1126 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2c2873572de4889a16fda8fe31ad594 consensus state has no leader
I20260501 13:57:27.780540 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 14b221e0725647df9789c2137ababefe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:27.787075 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_1af6dd9e62f34f618b1a05c80e7e16bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:27.788894 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1af6dd9e62f34f618b1a05c80e7e16bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:27.800241 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_445251ee80624ea789b17eee4cec7834"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:27.802924 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_445251ee80624ea789b17eee4cec7834 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:27.784396 1899 ts_tablet_manager.cc:1929] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:27.810787 1899 log.cc:1199] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/bbd44eebeb0347b9ac3ecfbdfc209d14
I20260501 13:57:27.811429 1899 ts_tablet_manager.cc:1950] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:27.813899 1759 tablet_service.cc:1511] Processing CreateTablet for tablet d74cf3b82ecf4379a81a1e2ec39213c9 (DEFAULT_TABLE table=default.Table_e71908fa21b440cea454e850fa601a0a [id=9bf3fd03858a42a2ba91370283546ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.819440 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 670f40234b8f43d485027818ac9694c8 (DEFAULT_TABLE table=default.Table_76959fdf8538441aa44f9e120288cdab [id=7803add44edc41d98147ce2f7ba60ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.820785 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670f40234b8f43d485027818ac9694c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.835125 1758 tablet_service.cc:1511] Processing CreateTablet for tablet d2c2873572de4889a16fda8fe31ad594 (DEFAULT_TABLE table=default.Table_7d92177d90f641498ccbdef2223fa7e4 [id=b8901f06222746b8bf418391866b92cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.841414 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d74cf3b82ecf4379a81a1e2ec39213c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.855170 1132 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_f25950402847469599f123ae30ac49f2" } modify_external_catalogs: true
I20260501 13:57:27.855253 1458 tablet_service.cc:1511] Processing CreateTablet for tablet 215a6200189c40d6aaebd6f1889b88c1 (DEFAULT_TABLE table=default.Table_b34565681ce74d37bbb0ebc07baff2e9 [id=59162f26c9244b389d0fda77569be02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.855784 1132 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_f25950402847469599f123ae30ac49f2" } modify_external_catalogs: true
I20260501 13:57:27.856426 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215a6200189c40d6aaebd6f1889b88c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.856506 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 670f40234b8f43d485027818ac9694c8 (DEFAULT_TABLE table=default.Table_76959fdf8538441aa44f9e120288cdab [id=7803add44edc41d98147ce2f7ba60ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.864107 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670f40234b8f43d485027818ac9694c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.869467 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet bbd44eebeb0347b9ac3ecfbdfc209d14 (table default.Table_c9015fd7203240529f60abbaf148a94a [id=dcc16a2384604fe280c4b99229047c21]) successfully deleted
W20260501 13:57:27.871583 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 7704a273008b47da9b158e280c91f06d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7704a273008b47da9b158e280c91f06d already in progress: creating tablet
I20260501 13:57:27.873621 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c2873572de4889a16fda8fe31ad594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.876681 1913 tablet.cc:1724] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260501 13:57:27.880364 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4c1ccc461d624514a37044a124d8ad6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:27.881289 1914 tablet.cc:1724] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20260501 13:57:27.882217 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 14b221e0725647df9789c2137ababefe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14b221e0725647df9789c2137ababefe already in progress: creating tablet
I20260501 13:57:27.885270 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_d746091594f546cc8a653e0653623394" } modify_external_catalogs: true
I20260501 13:57:27.885733 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_d746091594f546cc8a653e0653623394" } modify_external_catalogs: true
W20260501 13:57:27.882035 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c1ccc461d624514a37044a124d8ad6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:27.893913 1881 tablet_bootstrap.cc:492] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:27.895355 1881 ts_tablet_manager.cc:1403] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.339s user 0.008s sys 0.009s
I20260501 13:57:27.910642 1883 ts_tablet_manager.cc:1434] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.288s user 0.010s sys 0.000s
I20260501 13:57:27.914995 1882 tablet_bootstrap.cc:492] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:27.915417 1882 ts_tablet_manager.cc:1403] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.365s user 0.011s sys 0.010s
I20260501 13:57:27.922549 1881 raft_consensus.cc:359] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.923765 1881 raft_consensus.cc:385] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260501 13:57:27.924201 1920 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.249s user 0.000s sys 0.001s
I20260501 13:57:27.924160 1881 raft_consensus.cc:740] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.924826 1881 consensus_queue.cc:260] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:27.932260 1881 ts_tablet_manager.cc:1434] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s
I20260501 13:57:27.948367 1882 raft_consensus.cc:359] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.949292 1882 raft_consensus.cc:385] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:27.956754 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 215a6200189c40d6aaebd6f1889b88c1 (DEFAULT_TABLE table=default.Table_b34565681ce74d37bbb0ebc07baff2e9 [id=59162f26c9244b389d0fda77569be02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.958027 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215a6200189c40d6aaebd6f1889b88c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.944334 1883 tablet_bootstrap.cc:492] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:27.967842 1914 tablet.cc:1719] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Skipping requested alter to schema version 3, tablet already version 3
I20260501 13:57:27.970434 1881 tablet_bootstrap.cc:492] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:27.958773 1898 ts_tablet_manager.cc:1929] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:27.974345 1898 log.cc:1199] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/bbd44eebeb0347b9ac3ecfbdfc209d14
I20260501 13:57:27.975549 1898 ts_tablet_manager.cc:1950] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:27.978762 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet bbd44eebeb0347b9ac3ecfbdfc209d14 (table default.Table_c9015fd7203240529f60abbaf148a94a [id=dcc16a2384604fe280c4b99229047c21]) successfully deleted
I20260501 13:57:27.980336 1898 tablet_replica.cc:333] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:27.981036 1898 raft_consensus.cc:2243] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:27.981480 1898 raft_consensus.cc:2272] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:27.976485 1882 raft_consensus.cc:740] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:27.992810 1882 consensus_queue.cc:260] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:27.993436 1459 tablet_service.cc:1511] Processing CreateTablet for tablet d5d4064154c744198fa438cd3e3b7288 (DEFAULT_TABLE table=default.Table_4e1d3a6887b14eceae55db865debc857 [id=0dc2b91768f84b4d93b18f304d31883e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:27.994453 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5d4064154c744198fa438cd3e3b7288. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.999852 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_b34565681ce74d37bbb0ebc07baff2e9" } new_table_name: "default.Table_818fd205f01f45deaaf30421db6a783d" modify_external_catalogs: true
W20260501 13:57:28.002151 1119 catalog_manager.cc:4969] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 14b221e0725647df9789c2137ababefe because the tablet was not found. No further retry: Not found: Tablet not found: 14b221e0725647df9789c2137ababefe
I20260501 13:57:28.009428 1883 tablet_bootstrap.cc:654] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.011049 1898 ts_tablet_manager.cc:1916] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.012754 1913 tablet.cc:1719] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Skipping requested alter to schema version 3, tablet already version 3
I20260501 13:57:28.018428 1763 tablet_service.cc:1511] Processing CreateTablet for tablet d5d4064154c744198fa438cd3e3b7288 (DEFAULT_TABLE table=default.Table_4e1d3a6887b14eceae55db865debc857 [id=0dc2b91768f84b4d93b18f304d31883e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.019690 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5d4064154c744198fa438cd3e3b7288. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.020633 1462 tablet_service.cc:1511] Processing CreateTablet for tablet cf342e482ca04727a10cf79519549b69 (DEFAULT_TABLE table=default.Table_b9fc1d60ce8f47738143b74f0a59edf8 [id=412abad469fd41d3ba02a3b0926e7578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.021692 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf342e482ca04727a10cf79519549b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.029219 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet cc255bd2d2374f96bec525417a48dbcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:26 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:28.030143 1563 leader_election.cc:343] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: cc255bd2d2374f96bec525417a48dbcf
I20260501 13:57:28.031100 1897 tablet_replica.cc:333] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:28.032266 1610 tablet_service.cc:1511] Processing CreateTablet for tablet d2c2873572de4889a16fda8fe31ad594 (DEFAULT_TABLE table=default.Table_7d92177d90f641498ccbdef2223fa7e4 [id=b8901f06222746b8bf418391866b92cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.033469 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c2873572de4889a16fda8fe31ad594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:27.986362 1881 tablet_bootstrap.cc:654] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.043030 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 215a6200189c40d6aaebd6f1889b88c1 (DEFAULT_TABLE table=default.Table_b34565681ce74d37bbb0ebc07baff2e9 [id=59162f26c9244b389d0fda77569be02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.045799 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215a6200189c40d6aaebd6f1889b88c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.050835 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7889b062d4464b39a2ce4ad2eef30dee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:28.052983 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7889b062d4464b39a2ce4ad2eef30dee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:28.054409 1132 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 871448e01d554cc999ffa3099afeaa83
I20260501 13:57:28.058013 1460 tablet_service.cc:1558] Processing DeleteTablet for tablet 871448e01d554cc999ffa3099afeaa83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:28.059090 1127 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 05378c49253142f28589d42a70235385
I20260501 13:57:28.062734 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 871448e01d554cc999ffa3099afeaa83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:28.065698 1883 tablet_bootstrap.cc:492] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:28.066141 1883 ts_tablet_manager.cc:1403] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.007s
I20260501 13:57:28.068503 1460 tablet_service.cc:1558] Processing DeleteTablet for tablet 05378c49253142f28589d42a70235385 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:28.069334 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 871448e01d554cc999ffa3099afeaa83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 871448e01d554cc999ffa3099afeaa83 already in progress: creating tablet
I20260501 13:57:28.072813 1901 raft_consensus.cc:493] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:28.075592 1758 tablet_service.cc:1511] Processing CreateTablet for tablet cf342e482ca04727a10cf79519549b69 (DEFAULT_TABLE table=default.Table_b9fc1d60ce8f47738143b74f0a59edf8 [id=412abad469fd41d3ba02a3b0926e7578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.081063 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf342e482ca04727a10cf79519549b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.084537 1881 tablet_bootstrap.cc:492] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:28.084956 1881 ts_tablet_manager.cc:1403] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.115s user 0.010s sys 0.001s
I20260501 13:57:28.073276 1901 raft_consensus.cc:515] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.088701 1901 leader_election.cc:290] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:28.088966 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7704a273008b47da9b158e280c91f06d" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:28.090374 1717 leader_election.cc:343] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: Tablet not RUNNING: STOPPED
I20260501 13:57:28.089465 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 05378c49253142f28589d42a70235385 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:28.091499 1899 tablet_replica.cc:333] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:28.092290 1899 raft_consensus.cc:2243] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:28.093290 1899 raft_consensus.cc:2272] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.096302 1899 ts_tablet_manager.cc:1916] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.109352 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_de009c8498fe43d8a34a119924faa8d6" } new_table_name: "default.Table_79262cb3701444fe8ad303b4bbe08238" modify_external_catalogs: true
I20260501 13:57:28.110169 1881 raft_consensus.cc:359] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.111452 1883 raft_consensus.cc:359] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.112210 1883 raft_consensus.cc:385] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.112309 1881 raft_consensus.cc:385] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.113758 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4a46a5a6851d4b6f81255cbc9f59d74f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:28.115468 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a46a5a6851d4b6f81255cbc9f59d74f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:28.120779 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 215a6200189c40d6aaebd6f1889b88c1 consensus state has no leader
I20260501 13:57:28.121412 1603 tablet_service.cc:1511] Processing CreateTablet for tablet cf342e482ca04727a10cf79519549b69 (DEFAULT_TABLE table=default.Table_b9fc1d60ce8f47738143b74f0a59edf8 [id=412abad469fd41d3ba02a3b0926e7578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.112881 1881 raft_consensus.cc:740] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.124193 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf342e482ca04727a10cf79519549b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.124456 1881 consensus_queue.cc:260] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.127486 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 7704a273008b47da9b158e280c91f06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:28.129366 1881 ts_tablet_manager.cc:1434] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260501 13:57:28.119823 1882 ts_tablet_manager.cc:1434] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.203s user 0.007s sys 0.000s
I20260501 13:57:28.133419 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_f3ef319b6f34419d987e039dd1c2eb54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:28.134168 1604 tablet_service.cc:1511] Processing CreateTablet for tablet d5d4064154c744198fa438cd3e3b7288 (DEFAULT_TABLE table=default.Table_4e1d3a6887b14eceae55db865debc857 [id=0dc2b91768f84b4d93b18f304d31883e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.074134 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 670f40234b8f43d485027818ac9694c8 (DEFAULT_TABLE table=default.Table_76959fdf8538441aa44f9e120288cdab [id=7803add44edc41d98147ce2f7ba60ab4]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:28.136332 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3ef319b6f34419d987e039dd1c2eb54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:28.112479 1883 raft_consensus.cc:740] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.137420 1883 consensus_queue.cc:260] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.144001 1881 tablet_bootstrap.cc:492] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:28.145539 1898 ts_tablet_manager.cc:1929] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:28.145923 1898 log.cc:1199] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/7704a273008b47da9b158e280c91f06d
I20260501 13:57:28.146569 1898 ts_tablet_manager.cc:1950] T 7704a273008b47da9b158e280c91f06d P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:28.151068 1898 tablet_replica.cc:333] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:28.151830 1898 raft_consensus.cc:2243] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.152266 1898 raft_consensus.cc:2272] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.153146 1883 ts_tablet_manager.cc:1434] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.087s user 0.006s sys 0.001s
I20260501 13:57:28.149971 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 7704a273008b47da9b158e280c91f06d (table default.Table_26aa93a946694ca2b789c7eb59dc60c2 [id=6b6cfcedadc0495d9936a8624b7f62f7]) successfully deleted
I20260501 13:57:28.154852 1898 ts_tablet_manager.cc:1916] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:28.155390 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 14b221e0725647df9789c2137ababefe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14b221e0725647df9789c2137ababefe already in progress: creating tablet
I20260501 13:57:28.156250 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 495f5f95791b4817b50a60efc50fb30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:26 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:28.160885 1760 tablet_service.cc:1558] Processing DeleteTablet for tablet 14b221e0725647df9789c2137ababefe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:28.159927 1897 raft_consensus.cc:2243] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.162050 1897 raft_consensus.cc:2272] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.164669 1897 ts_tablet_manager.cc:1916] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.183571 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet 05378c49253142f28589d42a70235385 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:28.184304 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 871448e01d554cc999ffa3099afeaa83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:28.187204 1564 leader_election.cc:343] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Not found: Tablet not found: cc255bd2d2374f96bec525417a48dbcf
W20260501 13:57:28.187835 1126 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36e15d66bf6147a08604daf2a9c53322 has no consensus state
I20260501 13:57:28.187713 1564 leader_election.cc:304] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:28.189225 1564 raft_consensus.cc:2686] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260501 13:57:28.196183 1881 tablet_bootstrap.cc:654] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.204248 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5d4064154c744198fa438cd3e3b7288. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.204684 1898 ts_tablet_manager.cc:1929] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:28.205067 1898 log.cc:1199] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/871448e01d554cc999ffa3099afeaa83
I20260501 13:57:28.205699 1898 ts_tablet_manager.cc:1950] T 871448e01d554cc999ffa3099afeaa83 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:28.205873 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670f40234b8f43d485027818ac9694c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.208357 1898 tablet_replica.cc:333] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:28.208308 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 871448e01d554cc999ffa3099afeaa83 (table default.Table_f25950402847469599f123ae30ac49f2 [id=315879bd9f894b148f513e0b72d1d435]) successfully deleted
I20260501 13:57:28.209012 1898 raft_consensus.cc:2243] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.227766 1898 raft_consensus.cc:2272] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.230230 1898 ts_tablet_manager.cc:1916] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.241477 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7704a273008b47da9b158e280c91f06d" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:28.248225 1719 leader_election.cc:343] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:28.248663 1719 leader_election.cc:304] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:28.249975 1901 raft_consensus.cc:2749] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:28.251915 1125 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 3ca39bdb92b44986a6ffabbe2d33434c (table default.Table_c79a5c7717954055a8ff413892d23933 [id=9c8ebe4a498847a5a88c42e16f49d510]). Expected version 1 got 0
I20260501 13:57:28.259980 1898 ts_tablet_manager.cc:1929] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260501 13:57:28.260339 1898 log.cc:1199] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/05378c49253142f28589d42a70235385
I20260501 13:57:28.261154 1898 ts_tablet_manager.cc:1950] T 05378c49253142f28589d42a70235385 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:28.263631 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 05378c49253142f28589d42a70235385 (table default.Table_d746091594f546cc8a653e0653623394 [id=870dc36eb99c4cba874be3125bda1ca9]) successfully deleted
I20260501 13:57:28.265158 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_764f689a7c8b4b1e9791efa01e15286f" } modify_external_catalogs: true
I20260501 13:57:28.265681 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_764f689a7c8b4b1e9791efa01e15286f" } modify_external_catalogs: true
I20260501 13:57:28.265826 1128 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c79a5c7717954055a8ff413892d23933" } modify_external_catalogs: true
I20260501 13:57:28.266278 1128 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c79a5c7717954055a8ff413892d23933" } modify_external_catalogs: true
I20260501 13:57:28.273350 1881 tablet_bootstrap.cc:492] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:28.273756 1881 ts_tablet_manager.cc:1403] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.130s user 0.009s sys 0.003s
I20260501 13:57:28.276023 1881 raft_consensus.cc:359] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.276598 1881 raft_consensus.cc:385] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.276852 1881 raft_consensus.cc:740] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.279680 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 45907da8b56f4fe887253c77ae12f9a3 (DEFAULT_TABLE table=default.Table_1af6dd9e62f34f618b1a05c80e7e16bd [id=5cd8cae1c55f41aa809960d1cecddec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.280737 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45907da8b56f4fe887253c77ae12f9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.280660 1462 tablet_service.cc:1511] Processing CreateTablet for tablet b8d0f6d1f38b428291dba22015ac6959 (DEFAULT_TABLE table=default.Table_445251ee80624ea789b17eee4cec7834 [id=a9e4d906858945bbb1814b12e4c346ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.281670 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d0f6d1f38b428291dba22015ac6959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.283842 1887 raft_consensus.cc:493] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:28.284122 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 45907da8b56f4fe887253c77ae12f9a3 (DEFAULT_TABLE table=default.Table_1af6dd9e62f34f618b1a05c80e7e16bd [id=5cd8cae1c55f41aa809960d1cecddec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.284266 1887 raft_consensus.cc:515] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:28.285130 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45907da8b56f4fe887253c77ae12f9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.287744 1881 consensus_queue.cc:260] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.288183 1458 tablet_service.cc:1511] Processing CreateTablet for tablet 36e15d66bf6147a08604daf2a9c53322 (DEFAULT_TABLE table=default.Table_79262cb3701444fe8ad303b4bbe08238 [id=db653ae15d0b4dada66cc4a7b2ac94bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.289127 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e15d66bf6147a08604daf2a9c53322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.289224 1602 tablet_service.cc:1511] Processing CreateTablet for tablet b8d0f6d1f38b428291dba22015ac6959 (DEFAULT_TABLE table=default.Table_445251ee80624ea789b17eee4cec7834 [id=a9e4d906858945bbb1814b12e4c346ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.291216 1759 tablet_service.cc:1511] Processing CreateTablet for tablet b8d0f6d1f38b428291dba22015ac6959 (DEFAULT_TABLE table=default.Table_445251ee80624ea789b17eee4cec7834 [id=a9e4d906858945bbb1814b12e4c346ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.291669 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 36e15d66bf6147a08604daf2a9c53322 (DEFAULT_TABLE table=default.Table_79262cb3701444fe8ad303b4bbe08238 [id=db653ae15d0b4dada66cc4a7b2ac94bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.292260 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d0f6d1f38b428291dba22015ac6959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.292804 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e15d66bf6147a08604daf2a9c53322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.299011 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d0f6d1f38b428291dba22015ac6959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.304726 1887 leader_election.cc:290] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:28.310220 1882 tablet_bootstrap.cc:492] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:28.324463 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_1af6dd9e62f34f618b1a05c80e7e16bd" } new_table_name: "default.Table_4275f119eec547c28a46b8d5f4faa0f8" modify_external_catalogs: true
I20260501 13:57:28.332916 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 45907da8b56f4fe887253c77ae12f9a3 (DEFAULT_TABLE table=default.Table_1af6dd9e62f34f618b1a05c80e7e16bd [id=5cd8cae1c55f41aa809960d1cecddec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.334064 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45907da8b56f4fe887253c77ae12f9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.342284 1883 tablet_bootstrap.cc:492] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:28.342662 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca39bdb92b44986a6ffabbe2d33434c" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:28.343364 1783 raft_consensus.cc:2468] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:28.344822 1418 leader_election.cc:304] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:28.348351 1887 raft_consensus.cc:2804] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:28.348667 1887 raft_consensus.cc:493] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:28.348959 1887 raft_consensus.cc:3060] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:28.367018 1128 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 3ca39bdb92b44986a6ffabbe2d33434c
I20260501 13:57:28.369956 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 36e15d66bf6147a08604daf2a9c53322 (DEFAULT_TABLE table=default.Table_79262cb3701444fe8ad303b4bbe08238 [id=db653ae15d0b4dada66cc4a7b2ac94bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.372285 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ca39bdb92b44986a6ffabbe2d33434c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:28.372953 1898 tablet_replica.cc:333] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:28.373677 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4bb39ebdce2145ccaf94f17bb84197b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:28.382062 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bb39ebdce2145ccaf94f17bb84197b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:28.386690 1127 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 14fba31b7fdd42818f727fd2a745908d
W20260501 13:57:28.388768 1125 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45907da8b56f4fe887253c77ae12f9a3 consensus state has no leader
I20260501 13:57:28.372452 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e15d66bf6147a08604daf2a9c53322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.391201 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ca39bdb92b44986a6ffabbe2d33434c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:28.393690 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 14fba31b7fdd42818f727fd2a745908d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:28.398828 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 14fba31b7fdd42818f727fd2a745908d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:28.393596 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_1e50fec4e84b4259b65f9ce8e5684c48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:28.401436 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e50fec4e84b4259b65f9ce8e5684c48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:28.405292 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4e1d3a6887b14eceae55db865debc857" } new_table_name: "default.Table_f2b43b1500494a4bac22cfa9b776b136" modify_external_catalogs: true
I20260501 13:57:28.422670 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_cf979956be62407fabcf8e3f1c5f7f09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:28.424270 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf979956be62407fabcf8e3f1c5f7f09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:28.428443 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7e0182c80e7642909ee1d79a26b66cde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:28.430017 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e0182c80e7642909ee1d79a26b66cde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:28.443632 1920 tablet.cc:1724] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260501 13:57:28.445511 1887 raft_consensus.cc:515] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:28.438455 1882 tablet_bootstrap.cc:654] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.456930 1898 raft_consensus.cc:2243] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.457248 1887 leader_election.cc:290] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:28.458367 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca39bdb92b44986a6ffabbe2d33434c" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:28.458936 1783 raft_consensus.cc:3060] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:28.458951 1898 raft_consensus.cc:2272] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.461975 1898 ts_tablet_manager.cc:1916] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:28.462548 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5d4064154c744198fa438cd3e3b7288 consensus state has no leader
I20260501 13:57:28.467778 1783 raft_consensus.cc:2468] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:28.469233 1418 leader_election.cc:304] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters:
W20260501 13:57:28.469841 1418 raft_consensus.cc:2686] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260501 13:57:28.473488 1900 raft_consensus.cc:493] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:28.473924 1900 raft_consensus.cc:515] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.475646 1900 leader_election.cc:290] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:28.486063 1899 ts_tablet_manager.cc:1929] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260501 13:57:28.486483 1899 log.cc:1199] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/05378c49253142f28589d42a70235385
I20260501 13:57:28.487217 1899 ts_tablet_manager.cc:1950] T 05378c49253142f28589d42a70235385 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:28.489816 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 05378c49253142f28589d42a70235385 (table default.Table_d746091594f546cc8a653e0653623394 [id=870dc36eb99c4cba874be3125bda1ca9]) successfully deleted
I20260501 13:57:28.490557 1883 tablet_bootstrap.cc:654] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.497725 1883 tablet_bootstrap.cc:492] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:28.498107 1883 ts_tablet_manager.cc:1403] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.156s user 0.003s sys 0.006s
I20260501 13:57:28.502490 1758 tablet_service.cc:1511] Processing CreateTablet for tablet fc6b0cfd20174d7680f8981bb850c50f (DEFAULT_TABLE table=default.Table_f3ef319b6f34419d987e039dd1c2eb54 [id=c7df1385355c46c0896e9586953edf34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.503077 1899 tablet_replica.cc:333] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:28.502948 1759 tablet_service.cc:1511] Processing CreateTablet for tablet f2b59b51d81249e0ad90edba16803285 (DEFAULT_TABLE table=default.Table_7889b062d4464b39a2ce4ad2eef30dee [id=0444fad8c66840f2a003291fa6cc7208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.503149 1459 tablet_service.cc:1511] Processing CreateTablet for tablet fc6b0cfd20174d7680f8981bb850c50f (DEFAULT_TABLE table=default.Table_f3ef319b6f34419d987e039dd1c2eb54 [id=c7df1385355c46c0896e9586953edf34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.503566 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6b0cfd20174d7680f8981bb850c50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.503786 1899 raft_consensus.cc:2243] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.504122 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6b0cfd20174d7680f8981bb850c50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.504215 1899 raft_consensus.cc:2272] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.504391 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b59b51d81249e0ad90edba16803285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.505944 1899 ts_tablet_manager.cc:1916] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.506270 1763 tablet_service.cc:1511] Processing CreateTablet for tablet bf96cb4696704478a2064d6ce785d49d (DEFAULT_TABLE table=default.Table_4c1ccc461d624514a37044a124d8ad6b [id=314708984ce445e2a57e799bc77d5c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.506919 1883 raft_consensus.cc:359] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.507191 1460 tablet_service.cc:1511] Processing CreateTablet for tablet bf96cb4696704478a2064d6ce785d49d (DEFAULT_TABLE table=default.Table_4c1ccc461d624514a37044a124d8ad6b [id=314708984ce445e2a57e799bc77d5c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.507493 1883 raft_consensus.cc:385] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.507723 1883 raft_consensus.cc:740] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.508149 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf96cb4696704478a2064d6ce785d49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.509997 1883 consensus_queue.cc:260] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.512794 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 89e2c22052d04fe1a21bcb3323385111 (DEFAULT_TABLE table=default.Table_4a46a5a6851d4b6f81255cbc9f59d74f [id=36e8cb1c80cc45e7b4c501608c374af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.513840 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e2c22052d04fe1a21bcb3323385111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.521338 1883 ts_tablet_manager.cc:1434] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.023s user 0.004s sys 0.002s
I20260501 13:57:28.526732 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf96cb4696704478a2064d6ce785d49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.531071 1883 tablet_bootstrap.cc:492] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:28.549199 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 89e2c22052d04fe1a21bcb3323385111 (DEFAULT_TABLE table=default.Table_4a46a5a6851d4b6f81255cbc9f59d74f [id=36e8cb1c80cc45e7b4c501608c374af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.550246 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e2c22052d04fe1a21bcb3323385111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.555789 1883 tablet_bootstrap.cc:654] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.539212 1462 tablet_service.cc:1511] Processing CreateTablet for tablet f2b59b51d81249e0ad90edba16803285 (DEFAULT_TABLE table=default.Table_7889b062d4464b39a2ce4ad2eef30dee [id=0444fad8c66840f2a003291fa6cc7208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.567543 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b59b51d81249e0ad90edba16803285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.573215 1883 tablet_bootstrap.cc:492] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:28.573594 1883 ts_tablet_manager.cc:1403] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.002s
I20260501 13:57:28.574844 1125 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4c1ccc461d624514a37044a124d8ad6b" } modify_external_catalogs: true
I20260501 13:57:28.575379 1125 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4c1ccc461d624514a37044a124d8ad6b" } modify_external_catalogs: true
I20260501 13:57:28.575759 1883 raft_consensus.cc:359] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.576342 1883 raft_consensus.cc:385] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.576608 1883 raft_consensus.cc:740] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.580229 1899 ts_tablet_manager.cc:1929] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:28.582861 1899 log.cc:1199] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/7704a273008b47da9b158e280c91f06d
I20260501 13:57:28.583482 1899 ts_tablet_manager.cc:1950] T 7704a273008b47da9b158e280c91f06d P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:28.585871 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 7704a273008b47da9b158e280c91f06d (table default.Table_26aa93a946694ca2b789c7eb59dc60c2 [id=6b6cfcedadc0495d9936a8624b7f62f7]) successfully deleted
I20260501 13:57:28.587826 1883 consensus_queue.cc:260] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.563194 1898 ts_tablet_manager.cc:1929] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:28.589804 1898 log.cc:1199] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/3ca39bdb92b44986a6ffabbe2d33434c
I20260501 13:57:28.590597 1898 ts_tablet_manager.cc:1950] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:28.594489 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_7889b062d4464b39a2ce4ad2eef30dee" } modify_external_catalogs: true
I20260501 13:57:28.594595 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_1e50fec4e84b4259b65f9ce8e5684c48" } new_table_name: "default.Table_66545ca4c3d540ada57273201e17f2d8" modify_external_catalogs: true
I20260501 13:57:28.595065 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_7889b062d4464b39a2ce4ad2eef30dee" } modify_external_catalogs: true
W20260501 13:57:28.604859 1116 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_f2b43b1500494a4bac22cfa9b776b136 [id=0dc2b91768f84b4d93b18f304d31883e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d5d4064154c744198fa438cd3e3b7288 consensus state has no leader
W20260501 13:57:28.606928 1917 log.cc:927] Time spent T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Append to log took a long time: real 0.163s user 0.001s sys 0.000s
I20260501 13:57:28.623502 1881 ts_tablet_manager.cc:1434] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.349s user 0.009s sys 0.001s
I20260501 13:57:28.633985 1881 tablet_bootstrap.cc:492] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
W20260501 13:57:28.611485 1563 leader_election.cc:343] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 495f5f95791b4817b50a60efc50fb30c
W20260501 13:57:28.650244 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 43479.2us
I20260501 13:57:28.653112 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca39bdb92b44986a6ffabbe2d33434c" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:28.653999 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca39bdb92b44986a6ffabbe2d33434c" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:28.664562 1920 tablet.cc:1719] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Skipping requested alter to schema version 3, tablet already version 3
I20260501 13:57:28.665959 1897 ts_tablet_manager.cc:1929] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:28.666352 1897 log.cc:1199] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/cc255bd2d2374f96bec525417a48dbcf
I20260501 13:57:28.667287 1897 ts_tablet_manager.cc:1950] T cc255bd2d2374f96bec525417a48dbcf P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:28.670145 1897 tablet_replica.cc:333] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:28.670815 1897 raft_consensus.cc:2243] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.671257 1897 raft_consensus.cc:2272] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.673334 1897 ts_tablet_manager.cc:1916] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.677760 1882 tablet_bootstrap.cc:492] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:28.678177 1882 ts_tablet_manager.cc:1403] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.376s user 0.010s sys 0.008s
I20260501 13:57:28.680830 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_562821290a164c60b67184614e0d5580"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:28.680778 1882 raft_consensus.cc:359] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:28.681424 1882 raft_consensus.cc:385] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.681686 1882 raft_consensus.cc:740] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.683353 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ca39bdb92b44986a6ffabbe2d33434c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:28.683683 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4a46a5a6851d4b6f81255cbc9f59d74f" } new_table_name: "default.Table_4b9a916d6a754ac7b38819b39d0b1671" modify_external_catalogs: true
I20260501 13:57:28.684504 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 14fba31b7fdd42818f727fd2a745908d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:28.685922 1604 tablet_service.cc:1511] Processing CreateTablet for tablet fc6b0cfd20174d7680f8981bb850c50f (DEFAULT_TABLE table=default.Table_f3ef319b6f34419d987e039dd1c2eb54 [id=c7df1385355c46c0896e9586953edf34]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:28.682395 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_562821290a164c60b67184614e0d5580 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:28.686970 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6b0cfd20174d7680f8981bb850c50f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:28.692046 1119 consensus_peers.cc:597] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b -> Peer 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815): Couldn't send request to peer 50c0932256e74399a537670bb424a1b7. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260501 13:57:28.692320 1125 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet bf96cb4696704478a2064d6ce785d49d
I20260501 13:57:28.695111 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet bf96cb4696704478a2064d6ce785d49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:28.700289 1609 tablet_service.cc:1511] Processing CreateTablet for tablet bf96cb4696704478a2064d6ce785d49d (DEFAULT_TABLE table=default.Table_4c1ccc461d624514a37044a124d8ad6b [id=314708984ce445e2a57e799bc77d5c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.701363 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf96cb4696704478a2064d6ce785d49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.704085 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_c827da409c6d41dfb7eca61fcf4afa12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:28.705695 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c827da409c6d41dfb7eca61fcf4afa12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:28.707867 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c2d5336bfec43dbb38ce7071834bca3 has no consensus state
I20260501 13:57:28.708127 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 89e2c22052d04fe1a21bcb3323385111 (DEFAULT_TABLE table=default.Table_4a46a5a6851d4b6f81255cbc9f59d74f [id=36e8cb1c80cc45e7b4c501608c374af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.708984 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet f2b59b51d81249e0ad90edba16803285
I20260501 13:57:28.709132 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e2c22052d04fe1a21bcb3323385111. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:28.709448 1421 leader_election.cc:343] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:28.706444 1897 ts_tablet_manager.cc:1929] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260501 13:57:28.710420 1421 leader_election.cc:343] T 3ca39bdb92b44986a6ffabbe2d33434c P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:28.710605 1897 log.cc:1199] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/495f5f95791b4817b50a60efc50fb30c
I20260501 13:57:28.711452 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 3ca39bdb92b44986a6ffabbe2d33434c (table default.Table_c79a5c7717954055a8ff413892d23933 [id=9c8ebe4a498847a5a88c42e16f49d510]) successfully deleted
W20260501 13:57:28.712088 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 14fba31b7fdd42818f727fd2a745908d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14fba31b7fdd42818f727fd2a745908d already in progress: creating tablet
I20260501 13:57:28.711221 1897 ts_tablet_manager.cc:1950] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:28.714109 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet f2b59b51d81249e0ad90edba16803285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:28.714907 1897 tablet_replica.cc:333] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
W20260501 13:57:28.715325 1117 catalog_manager.cc:4969] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet f2b59b51d81249e0ad90edba16803285 because the tablet was not found. No further retry: Not found: Tablet not found: f2b59b51d81249e0ad90edba16803285
I20260501 13:57:28.715497 1897 raft_consensus.cc:2243] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.715989 1897 raft_consensus.cc:2272] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.718451 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet bf96cb4696704478a2064d6ce785d49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:28.720659 1897 ts_tablet_manager.cc:1916] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.723928 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_1476f68a25da45baa5470cbf3faea0a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:28.725556 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1476f68a25da45baa5470cbf3faea0a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:28.727185 1602 tablet_service.cc:1511] Processing CreateTablet for tablet f2b59b51d81249e0ad90edba16803285 (DEFAULT_TABLE table=default.Table_7889b062d4464b39a2ce4ad2eef30dee [id=0444fad8c66840f2a003291fa6cc7208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.728195 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b59b51d81249e0ad90edba16803285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.731535 1883 ts_tablet_manager.cc:1434] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.158s user 0.006s sys 0.000s
I20260501 13:57:28.734665 1901 raft_consensus.cc:493] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:28.735091 1901 raft_consensus.cc:515] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.736405 1883 tablet_bootstrap.cc:492] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:28.736711 1901 leader_election.cc:290] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:28.737695 1897 ts_tablet_manager.cc:1929] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260501 13:57:28.738085 1897 log.cc:1199] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/05378c49253142f28589d42a70235385
I20260501 13:57:28.738186 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14fba31b7fdd42818f727fd2a745908d" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:28.738910 1897 ts_tablet_manager.cc:1950] T 05378c49253142f28589d42a70235385 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:28.682271 1882 consensus_queue.cc:260] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:28.740789 1882 ts_tablet_manager.cc:1434] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.062s user 0.007s sys 0.000s
I20260501 13:57:28.741392 1899 tablet_replica.cc:333] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:28.742089 1899 raft_consensus.cc:2243] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.742547 1899 raft_consensus.cc:2272] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:28.743666 1717 leader_election.cc:343] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:28.744194 1899 ts_tablet_manager.cc:1916] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.746699 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet bf96cb4696704478a2064d6ce785d49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:28.748024 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_99fb9bd58f0d43cd9f314b8b3677a3f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:28.746330 1882 tablet_bootstrap.cc:492] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:28.749506 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet cc255bd2d2374f96bec525417a48dbcf (table default.Table_4099dfa8c6b84a9fb0fdb9fe6ed3b703 [id=ee3822ae2afd456fbb22e9860e37b240]) successfully deleted
W20260501 13:57:28.749612 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99fb9bd58f0d43cd9f314b8b3677a3f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:28.750379 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 495f5f95791b4817b50a60efc50fb30c (table default.Table_17e52f797589413ab3981e21e3a96045 [id=322103fbe9bb423999ba4858a447cbb4]) successfully deleted
I20260501 13:57:28.751169 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 05378c49253142f28589d42a70235385 (table default.Table_d746091594f546cc8a653e0653623394 [id=870dc36eb99c4cba874be3125bda1ca9]) successfully deleted
I20260501 13:57:28.752391 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14fba31b7fdd42818f727fd2a745908d" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:28.752759 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet bf96cb4696704478a2064d6ce785d49d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf96cb4696704478a2064d6ce785d49d already in progress: creating tablet
W20260501 13:57:28.752878 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 871448e01d554cc999ffa3099afeaa83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 871448e01d554cc999ffa3099afeaa83 already in progress: creating tablet
W20260501 13:57:28.753719 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 3ca39bdb92b44986a6ffabbe2d33434c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ca39bdb92b44986a6ffabbe2d33434c already in progress: creating tablet
W20260501 13:57:28.754575 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 14fba31b7fdd42818f727fd2a745908d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14fba31b7fdd42818f727fd2a745908d already in progress: creating tablet
I20260501 13:57:28.755046 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet f2b59b51d81249e0ad90edba16803285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:28.755528 1119 catalog_manager.cc:4969] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet bf96cb4696704478a2064d6ce785d49d because the tablet was not found. No further retry: Not found: Tablet not found: bf96cb4696704478a2064d6ce785d49d
W20260501 13:57:28.760952 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89e2c22052d04fe1a21bcb3323385111 consensus state has no leader
I20260501 13:57:28.765851 1883 tablet_bootstrap.cc:654] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.772699 1899 ts_tablet_manager.cc:1929] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:28.773100 1899 log.cc:1199] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/14b221e0725647df9789c2137ababefe
I20260501 13:57:28.773706 1899 ts_tablet_manager.cc:1950] T 14b221e0725647df9789c2137ababefe P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:28.778043 1882 tablet_bootstrap.cc:654] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:28.779748 1719 leader_election.cc:343] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:28.780205 1719 leader_election.cc:304] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:28.781680 1901 raft_consensus.cc:2749] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:28.783239 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet bbd44eebeb0347b9ac3ecfbdfc209d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:28.783901 1897 tablet_replica.cc:333] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:28.784485 1897 raft_consensus.cc:2243] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.784888 1897 raft_consensus.cc:2272] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.785693 1758 tablet_service.cc:1558] Processing DeleteTablet for tablet f2b59b51d81249e0ad90edba16803285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:28.786238 1119 catalog_manager.cc:4969] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet f2b59b51d81249e0ad90edba16803285 because the tablet was not found. No further retry: Not found: Tablet not found: f2b59b51d81249e0ad90edba16803285
I20260501 13:57:28.788496 1931 raft_consensus.cc:493] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:28.788942 1931 raft_consensus.cc:515] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:28.789286 1421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 78889.8us
I20260501 13:57:28.791553 1897 ts_tablet_manager.cc:1916] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:28.792606 1564 leader_election.cc:343] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Not found: Tablet not found: 495f5f95791b4817b50a60efc50fb30c
I20260501 13:57:28.793008 1564 leader_election.cc:304] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:28.793560 1564 raft_consensus.cc:2686] T 495f5f95791b4817b50a60efc50fb30c P 8f90b7ff170b4eadb27abc9ef866ffcf: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260501 13:57:28.776033 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 14b221e0725647df9789c2137ababefe (table default.Table_fe2abe8bde8846438f0be465b6c5cc2d [id=2d510b353dcc40938a2a8c3e34c99b76]) successfully deleted
I20260501 13:57:28.798290 1881 tablet_bootstrap.cc:654] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.806525 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "670f40234b8f43d485027818ac9694c8" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:28.807865 1421 leader_election.cc:343] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:28.809321 1899 tablet_replica.cc:333] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:28.811040 1899 raft_consensus.cc:2243] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.811540 1899 raft_consensus.cc:2272] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.813544 1899 ts_tablet_manager.cc:1916] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.822939 1931 leader_election.cc:290] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:28.823693 1882 tablet_bootstrap.cc:492] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:28.824155 1882 ts_tablet_manager.cc:1403] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.004s
I20260501 13:57:28.827284 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "670f40234b8f43d485027818ac9694c8" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:28.828222 1882 raft_consensus.cc:359] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
W20260501 13:57:28.828526 1418 leader_election.cc:343] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:28.828852 1882 raft_consensus.cc:385] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.828933 1418 leader_election.cc:304] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:28.829147 1882 raft_consensus.cc:740] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.829865 1931 raft_consensus.cc:2749] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:28.832288 1897 ts_tablet_manager.cc:1929] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:28.832636 1897 log.cc:1199] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/bbd44eebeb0347b9ac3ecfbdfc209d14
I20260501 13:57:28.833103 1883 tablet_bootstrap.cc:492] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:28.833257 1897 ts_tablet_manager.cc:1950] T bbd44eebeb0347b9ac3ecfbdfc209d14 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:28.833499 1883 ts_tablet_manager.cc:1403] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.097s user 0.009s sys 0.009s
I20260501 13:57:28.836233 1883 raft_consensus.cc:359] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.836814 1883 raft_consensus.cc:385] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.837046 1883 raft_consensus.cc:740] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.840098 1883 consensus_queue.cc:260] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.834765 1882 consensus_queue.cc:260] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } }
I20260501 13:57:28.842902 1881 tablet_bootstrap.cc:492] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:28.843302 1881 ts_tablet_manager.cc:1403] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.210s user 0.009s sys 0.004s
I20260501 13:57:28.845503 1881 raft_consensus.cc:359] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.846100 1881 raft_consensus.cc:385] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.849834 1882 ts_tablet_manager.cc:1434] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260501 13:57:28.850117 1883 ts_tablet_manager.cc:1434] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.016s user 0.001s sys 0.005s
I20260501 13:57:28.854878 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ca39bdb92b44986a6ffabbe2d33434c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:28.846342 1881 raft_consensus.cc:740] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.860145 1881 consensus_queue.cc:260] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.860765 1883 tablet_bootstrap.cc:492] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:28.861503 1882 tablet_bootstrap.cc:492] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:28.866345 1883 tablet_bootstrap.cc:654] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.868009 1882 tablet_bootstrap.cc:654] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.871708 1897 tablet_replica.cc:333] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:28.872372 1897 raft_consensus.cc:2243] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.872815 1897 raft_consensus.cc:2272] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.873533 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet bbd44eebeb0347b9ac3ecfbdfc209d14 (table default.Table_c9015fd7203240529f60abbaf148a94a [id=dcc16a2384604fe280c4b99229047c21]) successfully deleted
I20260501 13:57:28.875911 1881 ts_tablet_manager.cc:1434] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260501 13:57:28.877507 1899 ts_tablet_manager.cc:1929] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:28.877880 1899 log.cc:1199] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/3ca39bdb92b44986a6ffabbe2d33434c
I20260501 13:57:28.878441 1899 ts_tablet_manager.cc:1950] T 3ca39bdb92b44986a6ffabbe2d33434c P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:28.880298 1897 ts_tablet_manager.cc:1916] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.881532 1881 tablet_bootstrap.cc:492] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:28.884130 1883 tablet_bootstrap.cc:492] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:28.884523 1883 ts_tablet_manager.cc:1403] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20260501 13:57:28.885833 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet 14fba31b7fdd42818f727fd2a745908d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:28.886533 1898 tablet_replica.cc:333] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:28.886818 1883 raft_consensus.cc:359] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.887115 1898 raft_consensus.cc:2243] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.887379 1883 raft_consensus.cc:385] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.887507 1898 raft_consensus.cc:2272] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.887607 1883 raft_consensus.cc:740] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.888294 1899 tablet_replica.cc:333] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:28.888361 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 3ca39bdb92b44986a6ffabbe2d33434c (table default.Table_c79a5c7717954055a8ff413892d23933 [id=9c8ebe4a498847a5a88c42e16f49d510]) successfully deleted
I20260501 13:57:28.888742 1899 raft_consensus.cc:2243] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.889104 1899 raft_consensus.cc:2272] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.891955 1883 consensus_queue.cc:260] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.893556 1881 tablet_bootstrap.cc:654] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.895918 1898 ts_tablet_manager.cc:1916] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.898574 1899 ts_tablet_manager.cc:1916] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.898655 1883 ts_tablet_manager.cc:1434] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260501 13:57:28.899581 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_99fb9bd58f0d43cd9f314b8b3677a3f3" } modify_external_catalogs: true
I20260501 13:57:28.900199 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_99fb9bd58f0d43cd9f314b8b3677a3f3" } modify_external_catalogs: true
I20260501 13:57:28.905435 1883 tablet_bootstrap.cc:492] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:28.912773 1882 tablet_bootstrap.cc:492] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:28.913213 1882 ts_tablet_manager.cc:1403] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.003s
I20260501 13:57:28.916267 1899 ts_tablet_manager.cc:1929] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:28.916146 1125 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_562821290a164c60b67184614e0d5580" } modify_external_catalogs: true
I20260501 13:57:28.916599 1899 log.cc:1199] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/14fba31b7fdd42818f727fd2a745908d
I20260501 13:57:28.916671 1125 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_562821290a164c60b67184614e0d5580" } modify_external_catalogs: true
I20260501 13:57:28.916576 1882 raft_consensus.cc:359] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.917184 1899 ts_tablet_manager.cc:1950] T 14fba31b7fdd42818f727fd2a745908d P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:28.917222 1882 raft_consensus.cc:385] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.917479 1882 raft_consensus.cc:740] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.919489 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 14fba31b7fdd42818f727fd2a745908d (table default.Table_764f689a7c8b4b1e9791efa01e15286f [id=e4b3b630ee3a409f8efa7d0612d3a2f2]) successfully deleted
I20260501 13:57:28.920313 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_1476f68a25da45baa5470cbf3faea0a4" } new_table_name: "default.Table_26541c8dd6a347a78e26e6cb55fa3b06" modify_external_catalogs: true
W20260501 13:57:28.920991 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet bf96cb4696704478a2064d6ce785d49d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf96cb4696704478a2064d6ce785d49d already in progress: creating tablet
W20260501 13:57:28.922489 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet f2b59b51d81249e0ad90edba16803285 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2b59b51d81249e0ad90edba16803285 already in progress: creating tablet
I20260501 13:57:28.923305 1124 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_b6ea52503c0b4632a20c8e571d0d4ed8 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 1 got 0
I20260501 13:57:28.922442 1127 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet b1f387319f0748c3a61b1def05513902
I20260501 13:57:28.918121 1882 consensus_queue.cc:260] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.925223 1462 tablet_service.cc:1511] Processing CreateTablet for tablet c5297eb8aace46078cc3fce289fdd389 (DEFAULT_TABLE table=default.Table_7e0182c80e7642909ee1d79a26b66cde [id=37089fa463db4a608b09504153e2e32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:28.925933 1883 tablet_bootstrap.cc:654] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.926285 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5297eb8aace46078cc3fce289fdd389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:28.938094 1882 ts_tablet_manager.cc:1434] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20260501 13:57:28.941248 1124 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_b6ea52503c0b4632a20c8e571d0d4ed8 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 1 got 0
I20260501 13:57:28.942114 1124 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet d2c2873572de4889a16fda8fe31ad594 (table default.Table_7d92177d90f641498ccbdef2223fa7e4 [id=b8901f06222746b8bf418391866b92cc]). Expected version 1 got 0
I20260501 13:57:28.942832 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_57db8a12223e46c38099f51c4d3d370e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:28.943240 1882 tablet_bootstrap.cc:492] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
W20260501 13:57:28.944440 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57db8a12223e46c38099f51c4d3d370e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:28.949277 1897 ts_tablet_manager.cc:1929] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:28.949662 1897 log.cc:1199] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/3ca39bdb92b44986a6ffabbe2d33434c
I20260501 13:57:28.950275 1897 ts_tablet_manager.cc:1950] T 3ca39bdb92b44986a6ffabbe2d33434c P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:28.952515 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 3ca39bdb92b44986a6ffabbe2d33434c (table default.Table_c79a5c7717954055a8ff413892d23933 [id=9c8ebe4a498847a5a88c42e16f49d510]) successfully deleted
I20260501 13:57:28.957233 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 7704a273008b47da9b158e280c91f06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:28.958405 1882 tablet_bootstrap.cc:654] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:28.963707 1897 tablet_replica.cc:333] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:28.964339 1897 raft_consensus.cc:2243] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:28.964217 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c827da409c6d41dfb7eca61fcf4afa12" } new_table_name: "default.Table_9e3d29db155f4fd2ba6229fc2318edb0" modify_external_catalogs: true
I20260501 13:57:28.964776 1897 raft_consensus.cc:2272] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:28.969007 1897 ts_tablet_manager.cc:1916] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:28.969451 1125 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet b9caad4429b24488924523dcb2867286
I20260501 13:57:28.976677 1883 tablet_bootstrap.cc:492] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:28.977083 1883 ts_tablet_manager.cc:1403] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.005s
I20260501 13:57:28.979390 1883 raft_consensus.cc:359] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.979998 1883 raft_consensus.cc:385] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:28.980254 1883 raft_consensus.cc:740] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:28.980911 1883 consensus_queue.cc:260] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:28.982645 1883 ts_tablet_manager.cc:1434] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260501 13:57:28.983439 1883 tablet_bootstrap.cc:492] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:28.987618 1126 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ff05404061447199d936d005afda653 has no consensus state
I20260501 13:57:28.996119 1882 tablet_bootstrap.cc:492] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:28.996531 1882 ts_tablet_manager.cc:1403] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.004s
I20260501 13:57:28.994513 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_d35b1b0f1e60486eb80dc7c04849cb24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:28.999414 1882 raft_consensus.cc:359] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.000031 1882 raft_consensus.cc:385] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.000286 1882 raft_consensus.cc:740] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.000857 1882 consensus_queue.cc:260] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.003614 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet 871448e01d554cc999ffa3099afeaa83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:29.004716 1899 tablet_replica.cc:333] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:29.005352 1899 raft_consensus.cc:2243] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:29.005771 1899 raft_consensus.cc:2272] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:29.007225 1882 ts_tablet_manager.cc:1434] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
W20260501 13:57:29.007532 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a563bccc7b046658ed133109f676528 has no consensus state
I20260501 13:57:29.011109 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_396cced151574d61ba720d6f0115d6bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:29.012529 1882 tablet_bootstrap.cc:492] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:29.012796 1899 ts_tablet_manager.cc:1916] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:29.013378 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_396cced151574d61ba720d6f0115d6bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:29.015204 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d35b1b0f1e60486eb80dc7c04849cb24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.017854 1883 tablet_bootstrap.cc:654] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.019418 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_b6ea52503c0b4632a20c8e571d0d4ed8 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 1 got 0
I20260501 13:57:29.024974 1897 ts_tablet_manager.cc:1929] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:29.025342 1897 log.cc:1199] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/7704a273008b47da9b158e280c91f06d
I20260501 13:57:29.025933 1897 ts_tablet_manager.cc:1950] T 7704a273008b47da9b158e280c91f06d P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:29.026803 1882 tablet_bootstrap.cc:654] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.029181 1602 tablet_service.cc:1511] Processing CreateTablet for tablet c5297eb8aace46078cc3fce289fdd389 (DEFAULT_TABLE table=default.Table_7e0182c80e7642909ee1d79a26b66cde [id=37089fa463db4a608b09504153e2e32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.031150 1763 tablet_service.cc:1511] Processing CreateTablet for tablet c5297eb8aace46078cc3fce289fdd389 (DEFAULT_TABLE table=default.Table_7e0182c80e7642909ee1d79a26b66cde [id=37089fa463db4a608b09504153e2e32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.032246 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5297eb8aace46078cc3fce289fdd389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.034999 1883 tablet_bootstrap.cc:492] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:29.035387 1883 ts_tablet_manager.cc:1403] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.003s
I20260501 13:57:29.035687 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5297eb8aace46078cc3fce289fdd389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.039990 1882 tablet_bootstrap.cc:492] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:29.040380 1882 ts_tablet_manager.cc:1403] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.004s
I20260501 13:57:29.043155 1882 raft_consensus.cc:359] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.043742 1882 raft_consensus.cc:385] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.043987 1882 raft_consensus.cc:740] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.045382 1883 raft_consensus.cc:359] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.045987 1883 raft_consensus.cc:385] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.046231 1883 raft_consensus.cc:740] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.046888 1883 consensus_queue.cc:260] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.026137 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_2edf812ced814de8acadfe87bcad5893"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:29.048738 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2edf812ced814de8acadfe87bcad5893 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.049330 1883 ts_tablet_manager.cc:1434] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20260501 13:57:29.049852 1882 consensus_queue.cc:260] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.039162 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 7704a273008b47da9b158e280c91f06d (table default.Table_26aa93a946694ca2b789c7eb59dc60c2 [id=6b6cfcedadc0495d9936a8624b7f62f7]) successfully deleted
I20260501 13:57:29.054780 1883 tablet_bootstrap.cc:492] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:29.065377 1882 ts_tablet_manager.cc:1434] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260501 13:57:29.071244 1882 tablet_bootstrap.cc:492] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:29.080152 1458 tablet_service.cc:1511] Processing CreateTablet for tablet 8d2f196f44be4fbc8059901380b71271 (DEFAULT_TABLE table=default.Table_cf979956be62407fabcf8e3f1c5f7f09 [id=8cff6f6a2ebc447c979a45c9bfa438dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.081254 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2f196f44be4fbc8059901380b71271. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.081948 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 8d2f196f44be4fbc8059901380b71271 (DEFAULT_TABLE table=default.Table_cf979956be62407fabcf8e3f1c5f7f09 [id=8cff6f6a2ebc447c979a45c9bfa438dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.082504 1882 tablet_bootstrap.cc:654] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.083012 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2f196f44be4fbc8059901380b71271. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.084784 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 6c2d5336bfec43dbb38ce7071834bca3 (DEFAULT_TABLE table=default.Table_66545ca4c3d540ada57273201e17f2d8 [id=697eaa78d5e541988cd48466c64c5915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.085808 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c2d5336bfec43dbb38ce7071834bca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.086858 1883 tablet_bootstrap.cc:654] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.087718 1758 tablet_service.cc:1511] Processing CreateTablet for tablet 8d2f196f44be4fbc8059901380b71271 (DEFAULT_TABLE table=default.Table_cf979956be62407fabcf8e3f1c5f7f09 [id=8cff6f6a2ebc447c979a45c9bfa438dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.088784 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2f196f44be4fbc8059901380b71271. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.095854 1882 tablet_bootstrap.cc:492] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:29.096292 1882 ts_tablet_manager.cc:1403] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.000s
I20260501 13:57:29.100837 1883 tablet_bootstrap.cc:492] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:29.101285 1883 ts_tablet_manager.cc:1403] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.004s
I20260501 13:57:29.102770 1125 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_b6ea52503c0b4632a20c8e571d0d4ed8 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 1 got 0
I20260501 13:57:29.103816 1883 raft_consensus.cc:359] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.104409 1883 raft_consensus.cc:385] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.104669 1883 raft_consensus.cc:740] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.105382 1883 consensus_queue.cc:260] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.106273 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 14fba31b7fdd42818f727fd2a745908d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:29.107043 1897 tablet_replica.cc:333] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:29.107717 1897 raft_consensus.cc:2243] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:29.108153 1897 raft_consensus.cc:2272] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:29.108711 1882 raft_consensus.cc:359] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.109328 1882 raft_consensus.cc:385] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.109577 1882 raft_consensus.cc:740] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.113867 1883 ts_tablet_manager.cc:1434] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20260501 13:57:29.114768 1883 tablet_bootstrap.cc:492] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:29.119400 1898 ts_tablet_manager.cc:1929] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:29.119769 1898 log.cc:1199] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/14fba31b7fdd42818f727fd2a745908d
I20260501 13:57:29.120285 1883 tablet_bootstrap.cc:654] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.120343 1898 ts_tablet_manager.cc:1950] T 14fba31b7fdd42818f727fd2a745908d P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:29.121083 1901 raft_consensus.cc:493] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:29.121515 1901 raft_consensus.cc:515] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.123283 1901 leader_election.cc:290] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:29.124778 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c2873572de4889a16fda8fe31ad594" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:29.124912 1882 consensus_queue.cc:260] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:29.126626 1717 leader_election.cc:343] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:29.127714 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c2873572de4889a16fda8fe31ad594" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:29.128016 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4bb39ebdce2145ccaf94f17bb84197b8" } modify_external_catalogs: true
I20260501 13:57:29.128531 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4bb39ebdce2145ccaf94f17bb84197b8" } modify_external_catalogs: true
W20260501 13:57:29.129590 1719 leader_election.cc:343] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:29.130017 1719 leader_election.cc:304] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:29.131260 1897 ts_tablet_manager.cc:1916] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:29.131301 1901 raft_consensus.cc:2749] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:29.132833 1882 ts_tablet_manager.cc:1434] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20260501 13:57:29.136376 1882 tablet_bootstrap.cc:492] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:29.140236 1881 tablet_bootstrap.cc:492] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:29.140424 1124 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 215a6200189c40d6aaebd6f1889b88c1 (table default.Table_818fd205f01f45deaaf30421db6a783d [id=59162f26c9244b389d0fda77569be02a]). Expected version 1 got 0
I20260501 13:57:29.140779 1881 ts_tablet_manager.cc:1403] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.259s user 0.016s sys 0.006s
I20260501 13:57:29.143143 1881 raft_consensus.cc:359] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.143747 1881 raft_consensus.cc:385] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.143977 1881 raft_consensus.cc:740] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.148783 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 6c2d5336bfec43dbb38ce7071834bca3 (DEFAULT_TABLE table=default.Table_66545ca4c3d540ada57273201e17f2d8 [id=697eaa78d5e541988cd48466c64c5915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.149935 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c2d5336bfec43dbb38ce7071834bca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.151830 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 02926865010043e4b689c914b435400f (DEFAULT_TABLE table=default.Table_4bb39ebdce2145ccaf94f17bb84197b8 [id=a7c0d6e58eac4f9aabd0f2bd585db9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.152887 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02926865010043e4b689c914b435400f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.159618 1882 tablet_bootstrap.cc:654] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.163800 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 215a6200189c40d6aaebd6f1889b88c1 (table default.Table_818fd205f01f45deaaf30421db6a783d [id=59162f26c9244b389d0fda77569be02a]). Expected version 1 got 0
I20260501 13:57:29.166266 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 14fba31b7fdd42818f727fd2a745908d (table default.Table_764f689a7c8b4b1e9791efa01e15286f [id=e4b3b630ee3a409f8efa7d0612d3a2f2]) successfully deleted
I20260501 13:57:29.170897 1897 ts_tablet_manager.cc:1929] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:29.171288 1897 log.cc:1199] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/14fba31b7fdd42818f727fd2a745908d
I20260501 13:57:29.171912 1897 ts_tablet_manager.cc:1950] T 14fba31b7fdd42818f727fd2a745908d P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:29.174160 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 14fba31b7fdd42818f727fd2a745908d (table default.Table_764f689a7c8b4b1e9791efa01e15286f [id=e4b3b630ee3a409f8efa7d0612d3a2f2]) successfully deleted
I20260501 13:57:29.176419 1129 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_818fd205f01f45deaaf30421db6a783d" } modify_external_catalogs: true
I20260501 13:57:29.176990 1129 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_818fd205f01f45deaaf30421db6a783d" } modify_external_catalogs: true
I20260501 13:57:29.177567 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 6c2d5336bfec43dbb38ce7071834bca3 (DEFAULT_TABLE table=default.Table_66545ca4c3d540ada57273201e17f2d8 [id=697eaa78d5e541988cd48466c64c5915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.178261 1883 tablet_bootstrap.cc:492] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:29.178678 1883 ts_tablet_manager.cc:1403] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.064s user 0.010s sys 0.001s
I20260501 13:57:29.179412 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c2d5336bfec43dbb38ce7071834bca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.181703 1883 raft_consensus.cc:359] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.184345 1882 tablet_bootstrap.cc:492] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:29.184767 1882 ts_tablet_manager.cc:1403] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.006s
I20260501 13:57:29.185930 1881 consensus_queue.cc:260] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.186805 1758 tablet_service.cc:1511] Processing CreateTablet for tablet 02926865010043e4b689c914b435400f (DEFAULT_TABLE table=default.Table_4bb39ebdce2145ccaf94f17bb84197b8 [id=a7c0d6e58eac4f9aabd0f2bd585db9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.187161 1882 raft_consensus.cc:359] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.187768 1882 raft_consensus.cc:385] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.188023 1882 raft_consensus.cc:740] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.188181 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02926865010043e4b689c914b435400f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.182350 1883 raft_consensus.cc:385] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.188659 1883 raft_consensus.cc:740] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.189319 1883 consensus_queue.cc:260] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.191972 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_b6ea52503c0b4632a20c8e571d0d4ed8" } new_table_name: "default.Table_d4530066c45445778606ea4bbc21df36" modify_external_catalogs: true
I20260501 13:57:29.196364 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_abb70b9287914fbc97d3dcf78ac373bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:29.198094 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abb70b9287914fbc97d3dcf78ac373bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.199644 1883 ts_tablet_manager.cc:1434] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260501 13:57:29.199699 1881 ts_tablet_manager.cc:1434] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.059s user 0.005s sys 0.001s
I20260501 13:57:29.200281 1882 consensus_queue.cc:260] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.209053 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 02926865010043e4b689c914b435400f (DEFAULT_TABLE table=default.Table_4bb39ebdce2145ccaf94f17bb84197b8 [id=a7c0d6e58eac4f9aabd0f2bd585db9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.210935 1882 ts_tablet_manager.cc:1434] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.026s user 0.005s sys 0.002s
I20260501 13:57:29.211889 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_e71908fa21b440cea454e850fa601a0a" } new_table_name: "default.Table_734dfe3b26744b97baf53fd15e44c637" modify_external_catalogs: true
I20260501 13:57:29.217953 1883 tablet_bootstrap.cc:492] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:29.222072 1882 tablet_bootstrap.cc:492] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:29.227012 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02926865010043e4b689c914b435400f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.231652 1129 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 215a6200189c40d6aaebd6f1889b88c1
I20260501 13:57:29.233741 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 215a6200189c40d6aaebd6f1889b88c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:29.235505 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 215a6200189c40d6aaebd6f1889b88c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 215a6200189c40d6aaebd6f1889b88c1 already in progress: creating tablet
I20260501 13:57:29.237584 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 215a6200189c40d6aaebd6f1889b88c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:29.238687 1899 ts_tablet_manager.cc:1929] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:29.239115 1899 log.cc:1199] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/871448e01d554cc999ffa3099afeaa83
I20260501 13:57:29.240868 1899 ts_tablet_manager.cc:1950] T 871448e01d554cc999ffa3099afeaa83 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:29.243544 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 871448e01d554cc999ffa3099afeaa83 (table default.Table_f25950402847469599f123ae30ac49f2 [id=315879bd9f894b148f513e0b72d1d435]) successfully deleted
I20260501 13:57:29.245457 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 14b221e0725647df9789c2137ababefe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:29.246097 1897 tablet_replica.cc:333] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:29.246729 1897 raft_consensus.cc:2243] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:29.247133 1897 raft_consensus.cc:2272] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:29.249541 1897 ts_tablet_manager.cc:1916] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:29.258304 1883 tablet_bootstrap.cc:654] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.259485 1882 tablet_bootstrap.cc:654] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.269596 1900 raft_consensus.cc:493] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:29.270107 1900 raft_consensus.cc:515] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.271997 1900 leader_election.cc:290] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:29.272158 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 921e5c3eaaa44c62abbc90f560049614 consensus state has no leader
I20260501 13:57:29.273631 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "921e5c3eaaa44c62abbc90f560049614" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:29.274192 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "921e5c3eaaa44c62abbc90f560049614" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:29.274775 1783 raft_consensus.cc:2468] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
W20260501 13:57:29.275817 1564 leader_election.cc:343] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:29.276881 1563 leader_election.cc:304] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:29.277984 1132 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 45907da8b56f4fe887253c77ae12f9a3 (table default.Table_4275f119eec547c28a46b8d5f4faa0f8 [id=5cd8cae1c55f41aa809960d1cecddec3]). Expected version 1 got 0
I20260501 13:57:29.278054 1900 raft_consensus.cc:2804] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:29.278561 1900 raft_consensus.cc:493] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:29.278967 1900 raft_consensus.cc:3060] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:29.285526 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d74cf3b82ecf4379a81a1e2ec39213c9 consensus state has no leader
I20260501 13:57:29.289726 1900 raft_consensus.cc:515] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.292791 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "921e5c3eaaa44c62abbc90f560049614" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:29.293869 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "921e5c3eaaa44c62abbc90f560049614" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
W20260501 13:57:29.294217 1564 leader_election.cc:343] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:29.294478 1783 raft_consensus.cc:3060] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:29.295161 1882 tablet_bootstrap.cc:492] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:29.295549 1882 ts_tablet_manager.cc:1403] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.074s user 0.009s sys 0.000s
I20260501 13:57:29.296327 1458 tablet_service.cc:1558] Processing DeleteTablet for tablet 215a6200189c40d6aaebd6f1889b88c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:29.301342 1783 raft_consensus.cc:2468] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:29.303546 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4a57dcc1c3194af793a3a0367e766cc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:29.305461 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a57dcc1c3194af793a3a0367e766cc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:29.306775 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 215a6200189c40d6aaebd6f1889b88c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 215a6200189c40d6aaebd6f1889b88c1 already in progress: creating tablet
I20260501 13:57:29.311002 1882 raft_consensus.cc:359] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.311614 1882 raft_consensus.cc:385] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.311873 1882 raft_consensus.cc:740] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.311717 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_c6e79351ac8041739c99caa55b4af0ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:29.312422 1882 consensus_queue.cc:260] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:29.313342 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6e79351ac8041739c99caa55b4af0ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.316588 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_dc4bd9121deb4c58aadf6eb0d2282da6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:29.317147 1563 leader_election.cc:304] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:29.318339 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc4bd9121deb4c58aadf6eb0d2282da6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.318568 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 02926865010043e4b689c914b435400f
I20260501 13:57:29.320412 1458 tablet_service.cc:1558] Processing DeleteTablet for tablet 02926865010043e4b689c914b435400f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:29.322455 1117 catalog_manager.cc:4969] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 02926865010043e4b689c914b435400f because the tablet was not found. No further retry: Not found: Tablet not found: 02926865010043e4b689c914b435400f
I20260501 13:57:29.335417 1960 raft_consensus.cc:2804] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:29.335990 1960 raft_consensus.cc:697] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:29.336802 1758 tablet_service.cc:1558] Processing DeleteTablet for tablet 02926865010043e4b689c914b435400f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:29.334385 1900 leader_election.cc:290] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:29.348976 1960 consensus_queue.cc:237] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.353116 1881 tablet_bootstrap.cc:492] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:29.358325 1899 tablet_replica.cc:333] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:29.359083 1899 raft_consensus.cc:2243] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:29.359500 1899 raft_consensus.cc:2272] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:29.359640 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 02926865010043e4b689c914b435400f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:29.363605 1899 ts_tablet_manager.cc:1916] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:29.370294 1897 ts_tablet_manager.cc:1929] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:29.370729 1897 log.cc:1199] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/14b221e0725647df9789c2137ababefe
I20260501 13:57:29.371335 1897 ts_tablet_manager.cc:1950] T 14b221e0725647df9789c2137ababefe P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:29.383069 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 14b221e0725647df9789c2137ababefe (table default.Table_fe2abe8bde8846438f0be465b6c5cc2d [id=2d510b353dcc40938a2a8c3e34c99b76]) successfully deleted
I20260501 13:57:29.383649 1882 ts_tablet_manager.cc:1434] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.088s user 0.009s sys 0.000s
I20260501 13:57:29.384179 1881 tablet_bootstrap.cc:654] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.384509 1882 tablet_bootstrap.cc:492] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:29.390726 1899 ts_tablet_manager.cc:1929] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:29.391062 1899 log.cc:1199] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/215a6200189c40d6aaebd6f1889b88c1
I20260501 13:57:29.391630 1899 ts_tablet_manager.cc:1950] T 215a6200189c40d6aaebd6f1889b88c1 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
W20260501 13:57:29.392756 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 02926865010043e4b689c914b435400f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02926865010043e4b689c914b435400f already in progress: creating tablet
I20260501 13:57:29.393739 1882 tablet_bootstrap.cc:654] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.394385 1883 tablet_bootstrap.cc:492] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:29.394796 1883 ts_tablet_manager.cc:1403] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.178s user 0.018s sys 0.018s
I20260501 13:57:29.396962 1883 raft_consensus.cc:359] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.397552 1883 raft_consensus.cc:385] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.397814 1883 raft_consensus.cc:740] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.398423 1883 consensus_queue.cc:260] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.411103 1883 ts_tablet_manager.cc:1434] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260501 13:57:29.413096 1960 raft_consensus.cc:493] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:29.413554 1960 raft_consensus.cc:515] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.415369 1960 leader_election.cc:290] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:29.418218 1563 leader_election.cc:343] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 871448e01d554cc999ffa3099afeaa83
W20260501 13:57:29.418663 1564 leader_election.cc:343] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Not found: Tablet not found: 871448e01d554cc999ffa3099afeaa83
I20260501 13:57:29.419039 1564 leader_election.cc:304] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:29.419346 1883 tablet_bootstrap.cc:492] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:29.419903 1960 raft_consensus.cc:2749] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:29.420055 1127 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_79262cb3701444fe8ad303b4bbe08238 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 1 got 0
I20260501 13:57:29.428189 1881 tablet_bootstrap.cc:492] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:29.428615 1881 ts_tablet_manager.cc:1403] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.076s user 0.009s sys 0.000s
I20260501 13:57:29.431541 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 215a6200189c40d6aaebd6f1889b88c1 (table default.Table_818fd205f01f45deaaf30421db6a783d [id=59162f26c9244b389d0fda77569be02a]) successfully deleted
W20260501 13:57:29.433224 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 02926865010043e4b689c914b435400f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02926865010043e4b689c914b435400f already in progress: creating tablet
I20260501 13:57:29.435557 1127 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_79262cb3701444fe8ad303b4bbe08238 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 1 got 0
I20260501 13:57:29.436993 1883 tablet_bootstrap.cc:654] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.439414 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_734dfe3b26744b97baf53fd15e44c637" } modify_external_catalogs: true
I20260501 13:57:29.440014 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_734dfe3b26744b97baf53fd15e44c637" } modify_external_catalogs: true
I20260501 13:57:29.442732 1881 raft_consensus.cc:359] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.443392 1881 raft_consensus.cc:385] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.443643 1881 raft_consensus.cc:740] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.444203 1881 consensus_queue.cc:260] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.443917 1128 catalog_manager.cc:5671] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:29.445174 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_d4530066c45445778606ea4bbc21df36 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 2 got 0
I20260501 13:57:29.447124 1882 tablet_bootstrap.cc:492] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:29.447520 1882 ts_tablet_manager.cc:1403] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.003s
I20260501 13:57:29.449858 1882 raft_consensus.cc:359] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.457834 1883 tablet_bootstrap.cc:492] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:29.458230 1883 ts_tablet_manager.cc:1403] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.016s
I20260501 13:57:29.460539 1883 raft_consensus.cc:359] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.461138 1883 raft_consensus.cc:385] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.461387 1883 raft_consensus.cc:740] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.462000 1881 ts_tablet_manager.cc:1434] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260501 13:57:29.461943 1883 consensus_queue.cc:260] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.462877 1881 tablet_bootstrap.cc:492] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:29.463864 1883 ts_tablet_manager.cc:1434] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260501 13:57:29.465795 1883 tablet_bootstrap.cc:492] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:29.467577 1882 raft_consensus.cc:385] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.467900 1882 raft_consensus.cc:740] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.468600 1882 consensus_queue.cc:260] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.475739 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_7e0182c80e7642909ee1d79a26b66cde" } new_table_name: "default.Table_eb039fccfc7d444baf4e6cdddc6be5cd" modify_external_catalogs: true
I20260501 13:57:29.480322 1883 tablet_bootstrap.cc:654] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.488703 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_3fd69d2d471947c68f1becc5206a9c0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:29.490404 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fd69d2d471947c68f1becc5206a9c0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.492476 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_cf979956be62407fabcf8e3f1c5f7f09" } new_table_name: "default.Table_4b01fcca8d2a46ccba7050ab708d8a8d" modify_external_catalogs: true
I20260501 13:57:29.506908 1882 ts_tablet_manager.cc:1434] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20260501 13:57:29.510996 1126 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet d74cf3b82ecf4379a81a1e2ec39213c9 (table default.Table_734dfe3b26744b97baf53fd15e44c637 [id=9bf3fd03858a42a2ba91370283546ecf]). Expected version 1 got 0
I20260501 13:57:29.514109 1465 tablet_service.cc:1511] Processing CreateTablet for tablet f7645593dc184dab9f677f9371a88354 (DEFAULT_TABLE table=default.Table_d35b1b0f1e60486eb80dc7c04849cb24 [id=b2d6c0379e3d452593a7bf80f5877c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.515756 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7645593dc184dab9f677f9371a88354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.517225 1882 tablet_bootstrap.cc:492] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:29.520619 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 871448e01d554cc999ffa3099afeaa83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:29.521303 1897 tablet_replica.cc:333] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:29.522830 1604 tablet_service.cc:1511] Processing CreateTablet for tablet f7645593dc184dab9f677f9371a88354 (DEFAULT_TABLE table=default.Table_d35b1b0f1e60486eb80dc7c04849cb24 [id=b2d6c0379e3d452593a7bf80f5877c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.523819 1897 raft_consensus.cc:2243] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:29.524263 1897 raft_consensus.cc:2272] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:29.524555 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7645593dc184dab9f677f9371a88354. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:29.526944 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5297eb8aace46078cc3fce289fdd389 consensus state has no leader
I20260501 13:57:29.530215 1901 raft_consensus.cc:493] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:29.530673 1901 raft_consensus.cc:515] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.531419 1758 tablet_service.cc:1511] Processing CreateTablet for tablet f7645593dc184dab9f677f9371a88354 (DEFAULT_TABLE table=default.Table_d35b1b0f1e60486eb80dc7c04849cb24 [id=b2d6c0379e3d452593a7bf80f5877c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.532428 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7645593dc184dab9f677f9371a88354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.532555 1901 leader_election.cc:290] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:29.534061 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8d0f6d1f38b428291dba22015ac6959" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:29.536043 1717 leader_election.cc:343] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:29.535954 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4120cd7643bd4ca8bdd6b6c163c8e375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:29.538234 1763 tablet_service.cc:1511] Processing CreateTablet for tablet adc47b3b2b084d33a12bd28cfad27382 (DEFAULT_TABLE table=default.Table_57db8a12223e46c38099f51c4d3d370e [id=1de866dfd0d145e7a13e5ae1f367524b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.539211 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adc47b3b2b084d33a12bd28cfad27382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.540071 1132 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_dc4bd9121deb4c58aadf6eb0d2282da6" } modify_external_catalogs: true
I20260501 13:57:29.540607 1132 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_dc4bd9121deb4c58aadf6eb0d2282da6" } modify_external_catalogs: true
I20260501 13:57:29.541594 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 2ff05404061447199d936d005afda653 (DEFAULT_TABLE table=default.Table_26541c8dd6a347a78e26e6cb55fa3b06 [id=15dd9506496343159cc3f4de3afcef5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.542593 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ff05404061447199d936d005afda653. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:29.538420 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4120cd7643bd4ca8bdd6b6c163c8e375 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.546839 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8d0f6d1f38b428291dba22015ac6959" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:29.548182 1719 leader_election.cc:343] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:29.548586 1719 leader_election.cc:304] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:29.549530 1901 raft_consensus.cc:2749] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:29.551617 1762 tablet_service.cc:1511] Processing CreateTablet for tablet e41b43329e054442bc8697cc28161954 (DEFAULT_TABLE table=default.Table_2edf812ced814de8acadfe87bcad5893 [id=694ff2769ea04445811ca8d8a92854fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.552712 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41b43329e054442bc8697cc28161954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.556308 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_77b89c72ca0c4f278237ee79c17b6693"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:29.557476 1458 tablet_service.cc:1511] Processing CreateTablet for tablet adc47b3b2b084d33a12bd28cfad27382 (DEFAULT_TABLE table=default.Table_57db8a12223e46c38099f51c4d3d370e [id=1de866dfd0d145e7a13e5ae1f367524b]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:29.558034 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77b89c72ca0c4f278237ee79c17b6693 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.558552 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adc47b3b2b084d33a12bd28cfad27382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.569005 1609 tablet_service.cc:1511] Processing CreateTablet for tablet adc47b3b2b084d33a12bd28cfad27382 (DEFAULT_TABLE table=default.Table_57db8a12223e46c38099f51c4d3d370e [id=1de866dfd0d145e7a13e5ae1f367524b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.570083 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adc47b3b2b084d33a12bd28cfad27382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.574326 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 2ff05404061447199d936d005afda653 (DEFAULT_TABLE table=default.Table_26541c8dd6a347a78e26e6cb55fa3b06 [id=15dd9506496343159cc3f4de3afcef5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.575423 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ff05404061447199d936d005afda653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.575233 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 2ff05404061447199d936d005afda653 (DEFAULT_TABLE table=default.Table_26541c8dd6a347a78e26e6cb55fa3b06 [id=15dd9506496343159cc3f4de3afcef5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.576267 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ff05404061447199d936d005afda653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.584790 1459 tablet_service.cc:1511] Processing CreateTablet for tablet e41b43329e054442bc8697cc28161954 (DEFAULT_TABLE table=default.Table_2edf812ced814de8acadfe87bcad5893 [id=694ff2769ea04445811ca8d8a92854fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.585945 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41b43329e054442bc8697cc28161954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.590282 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 4a563bccc7b046658ed133109f676528 (DEFAULT_TABLE table=default.Table_9e3d29db155f4fd2ba6229fc2318edb0 [id=08c0033dd652465797b8be473d0e427f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.591441 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a563bccc7b046658ed133109f676528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.593483 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 5e86d6c935d6444b8a953e7be1bdfa51 (DEFAULT_TABLE table=default.Table_396cced151574d61ba720d6f0115d6bd [id=0f9c1230865848adaa4e5ea744c31c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.594561 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e86d6c935d6444b8a953e7be1bdfa51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.600255 1606 tablet_service.cc:1511] Processing CreateTablet for tablet e41b43329e054442bc8697cc28161954 (DEFAULT_TABLE table=default.Table_2edf812ced814de8acadfe87bcad5893 [id=694ff2769ea04445811ca8d8a92854fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.601337 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41b43329e054442bc8697cc28161954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.608477 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 4a563bccc7b046658ed133109f676528 (DEFAULT_TABLE table=default.Table_9e3d29db155f4fd2ba6229fc2318edb0 [id=08c0033dd652465797b8be473d0e427f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.609644 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a563bccc7b046658ed133109f676528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.613814 1603 tablet_service.cc:1511] Processing CreateTablet for tablet 5e86d6c935d6444b8a953e7be1bdfa51 (DEFAULT_TABLE table=default.Table_396cced151574d61ba720d6f0115d6bd [id=0f9c1230865848adaa4e5ea744c31c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.612785 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 4a563bccc7b046658ed133109f676528 (DEFAULT_TABLE table=default.Table_9e3d29db155f4fd2ba6229fc2318edb0 [id=08c0033dd652465797b8be473d0e427f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.615336 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a563bccc7b046658ed133109f676528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.615934 1460 tablet_service.cc:1511] Processing CreateTablet for tablet 5e86d6c935d6444b8a953e7be1bdfa51 (DEFAULT_TABLE table=default.Table_396cced151574d61ba720d6f0115d6bd [id=0f9c1230865848adaa4e5ea744c31c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.616994 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e86d6c935d6444b8a953e7be1bdfa51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.619375 1131 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_d4530066c45445778606ea4bbc21df36 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 2 got 0
I20260501 13:57:29.620220 1131 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet d2c2873572de4889a16fda8fe31ad594 (table default.Table_7d92177d90f641498ccbdef2223fa7e4 [id=b8901f06222746b8bf418391866b92cc]). Expected version 1 got 0
I20260501 13:57:29.625782 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e86d6c935d6444b8a953e7be1bdfa51. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:29.632256 1116 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 4a563bccc7b046658ed133109f676528 (table default.Table_9e3d29db155f4fd2ba6229fc2318edb0 [id=08c0033dd652465797b8be473d0e427f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4a563bccc7b046658ed133109f676528 consensus state has no leader
I20260501 13:57:29.633024 1881 tablet_bootstrap.cc:654] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.636883 1897 ts_tablet_manager.cc:1916] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:29.643592 1882 tablet_bootstrap.cc:654] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.644271 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_2294cbb031954b92a89e152ac50cb9f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:29.647532 1901 raft_consensus.cc:493] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:29.647989 1901 raft_consensus.cc:515] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:29.649168 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2294cbb031954b92a89e152ac50cb9f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.650421 1931 raft_consensus.cc:493] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:29.650867 1931 raft_consensus.cc:515] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.652673 1931 leader_election.cc:290] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:29.653623 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74cf3b82ecf4379a81a1e2ec39213c9" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:29.654333 1630 raft_consensus.cc:2468] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:29.655675 1421 leader_election.cc:304] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:29.656597 1931 raft_consensus.cc:2804] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:29.657246 1931 raft_consensus.cc:493] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:29.657613 1931 raft_consensus.cc:3060] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:29.664610 1931 raft_consensus.cc:515] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:29.665233 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d2f196f44be4fbc8059901380b71271 consensus state has no leader
I20260501 13:57:29.667989 1127 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet d74cf3b82ecf4379a81a1e2ec39213c9
I20260501 13:57:29.670209 1883 tablet_bootstrap.cc:492] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:29.670673 1883 ts_tablet_manager.cc:1403] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.205s user 0.025s sys 0.025s
I20260501 13:57:29.672844 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74cf3b82ecf4379a81a1e2ec39213c9" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:29.673386 1630 raft_consensus.cc:3060] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:29.674333 1757 tablet_service.cc:1558] Processing DeleteTablet for tablet d74cf3b82ecf4379a81a1e2ec39213c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:29.675105 1899 tablet_replica.cc:333] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:29.675772 1899 raft_consensus.cc:2243] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:29.676232 1899 raft_consensus.cc:2272] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:29.680718 1630 raft_consensus.cc:2468] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:29.681843 1421 leader_election.cc:304] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:29.688891 1931 leader_election.cc:290] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:29.689859 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet d74cf3b82ecf4379a81a1e2ec39213c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:29.690132 1931 raft_consensus.cc:2804] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:29.690699 1898 tablet_replica.cc:333] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:29.691907 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet 14b221e0725647df9789c2137ababefe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:27 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:29.691927 1931 raft_consensus.cc:697] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:29.702296 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet d74cf3b82ecf4379a81a1e2ec39213c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:29.704742 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "670f40234b8f43d485027818ac9694c8" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:29.705386 1485 raft_consensus.cc:2468] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 0.
I20260501 13:57:29.705652 1901 leader_election.cc:290] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:29.706439 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "670f40234b8f43d485027818ac9694c8" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:29.706518 1717 leader_election.cc:304] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:29.706712 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74cf3b82ecf4379a81a1e2ec39213c9" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:29.707295 1901 raft_consensus.cc:2804] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:29.707608 1901 raft_consensus.cc:493] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260501 13:57:29.707806 1418 leader_election.cc:343] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:29.707928 1901 raft_consensus.cc:3060] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:29.709332 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74cf3b82ecf4379a81a1e2ec39213c9" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
W20260501 13:57:29.710667 1719 leader_election.cc:343] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:29.711830 1418 leader_election.cc:343] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:29.712208 1899 ts_tablet_manager.cc:1916] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:29.713999 1883 raft_consensus.cc:359] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.714771 1883 raft_consensus.cc:385] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.715071 1883 raft_consensus.cc:740] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.715751 1883 consensus_queue.cc:260] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.726608 1132 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 1ecd2e71afd74f00af66408da8092ca6
I20260501 13:57:29.734931 1897 ts_tablet_manager.cc:1929] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:29.735311 1897 log.cc:1199] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/871448e01d554cc999ffa3099afeaa83
I20260501 13:57:29.735914 1897 ts_tablet_manager.cc:1950] T 871448e01d554cc999ffa3099afeaa83 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:29.738168 1897 tablet_replica.cc:333] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:29.738830 1897 raft_consensus.cc:2243] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:29.741499 1931 consensus_queue.cc:237] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.750494 1881 tablet_bootstrap.cc:492] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:29.750955 1881 ts_tablet_manager.cc:1403] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.288s user 0.011s sys 0.000s
I20260501 13:57:29.750990 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 871448e01d554cc999ffa3099afeaa83 (table default.Table_f25950402847469599f123ae30ac49f2 [id=315879bd9f894b148f513e0b72d1d435]) successfully deleted
I20260501 13:57:29.751842 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_396cced151574d61ba720d6f0115d6bd" } new_table_name: "default.Table_bda4446a8e3c4d5ab3d69dc4aeab8b4f" modify_external_catalogs: true
I20260501 13:57:29.753466 1881 raft_consensus.cc:359] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.754136 1881 raft_consensus.cc:385] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260501 13:57:29.755597 1421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 73844.8us
I20260501 13:57:29.741744 1783 raft_consensus.cc:1275] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Refusing update from remote peer 8f90b7ff170b4eadb27abc9ef866ffcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:57:29.760250 1960 consensus_queue.cc:1048] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:57:29.754376 1881 raft_consensus.cc:740] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.765450 1898 raft_consensus.cc:2243] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:29.766757 1898 pending_rounds.cc:70] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: Trying to abort 1 pending ops.
I20260501 13:57:29.767038 1898 pending_rounds.cc:77] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229208550961152 op_type: NO_OP noop_request { }
I20260501 13:57:29.767395 1898 raft_consensus.cc:2889] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:29.767712 1898 raft_consensus.cc:2272] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:29.768163 1881 consensus_queue.cc:260] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.770733 1881 ts_tablet_manager.cc:1434] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260501 13:57:29.773422 1901 raft_consensus.cc:515] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:29.775346 1974 consensus_queue.cc:1175] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "993b9ba18d8547f8af43167e727a6a24" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
W20260501 13:57:29.781304 1564 consensus_peers.cc:597] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:29.782039 1881 tablet_bootstrap.cc:492] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:29.786036 1883 ts_tablet_manager.cc:1434] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.115s user 0.013s sys 0.001s
I20260501 13:57:29.787547 1881 tablet_bootstrap.cc:654] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.790136 1898 ts_tablet_manager.cc:1916] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:29.804636 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "670f40234b8f43d485027818ac9694c8" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:29.805230 1485 raft_consensus.cc:3060] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:29.805555 1883 tablet_bootstrap.cc:492] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:29.807708 1897 raft_consensus.cc:2272] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:29.807924 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "670f40234b8f43d485027818ac9694c8" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:29.809912 1897 ts_tablet_manager.cc:1916] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:29.811355 1898 ts_tablet_manager.cc:1929] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:29.811715 1898 log.cc:1199] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/d74cf3b82ecf4379a81a1e2ec39213c9
I20260501 13:57:29.812342 1898 ts_tablet_manager.cc:1950] T d74cf3b82ecf4379a81a1e2ec39213c9 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:29.815165 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet d74cf3b82ecf4379a81a1e2ec39213c9 (table default.Table_734dfe3b26744b97baf53fd15e44c637 [id=9bf3fd03858a42a2ba91370283546ecf]) successfully deleted
I20260501 13:57:29.818579 1898 tablet_replica.cc:333] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:29.819320 1898 raft_consensus.cc:2243] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:29.819612 1485 raft_consensus.cc:2468] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 1.
I20260501 13:57:29.819872 1898 raft_consensus.cc:2272] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:29.802595 1901 leader_election.cc:290] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
W20260501 13:57:29.821786 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e86d6c935d6444b8a953e7be1bdfa51 consensus state has no leader
I20260501 13:57:29.822127 1898 ts_tablet_manager.cc:1916] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:29.831583 1881 tablet_bootstrap.cc:492] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:29.831977 1881 ts_tablet_manager.cc:1403] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.000s
I20260501 13:57:29.834227 1881 raft_consensus.cc:359] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.834888 1881 raft_consensus.cc:385] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.835131 1881 raft_consensus.cc:740] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
W20260501 13:57:29.836427 1719 leader_election.cc:343] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:29.836534 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet d74cf3b82ecf4379a81a1e2ec39213c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:29.839342 1717 leader_election.cc:304] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:29.840240 1897 ts_tablet_manager.cc:1929] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:29.840592 1897 log.cc:1199] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/d74cf3b82ecf4379a81a1e2ec39213c9
I20260501 13:57:29.843503 1128 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet d2c2873572de4889a16fda8fe31ad594 (table default.Table_7d92177d90f641498ccbdef2223fa7e4 [id=b8901f06222746b8bf418391866b92cc]). Expected version 1 got 0
I20260501 13:57:29.845403 1901 raft_consensus.cc:2804] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:29.841179 1897 ts_tablet_manager.cc:1950] T d74cf3b82ecf4379a81a1e2ec39213c9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:29.845841 1901 raft_consensus.cc:697] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Running, Role: LEADER
I20260501 13:57:29.835770 1881 consensus_queue.cc:260] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.847177 1901 consensus_queue.cc:237] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.854171 1128 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_d4530066c45445778606ea4bbc21df36 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 2 got 0
I20260501 13:57:29.857571 1465 tablet_service.cc:1511] Processing CreateTablet for tablet c33a1f423bdb44f5843daffe8f73e74a (DEFAULT_TABLE table=default.Table_c6e79351ac8041739c99caa55b4af0ff [id=d98d5437c5bc4e44bcfe27983dfa36e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.858448 1881 ts_tablet_manager.cc:1434] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
I20260501 13:57:29.859251 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c33a1f423bdb44f5843daffe8f73e74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.861109 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet d74cf3b82ecf4379a81a1e2ec39213c9 (table default.Table_734dfe3b26744b97baf53fd15e44c637 [id=9bf3fd03858a42a2ba91370283546ecf]) successfully deleted
I20260501 13:57:29.867030 1881 tablet_bootstrap.cc:492] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:29.868712 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_7d92177d90f641498ccbdef2223fa7e4" } new_table_name: "default.Table_3d853a6bfdc945fd9fb7c23468905983" modify_external_catalogs: true
I20260501 13:57:29.866286 1128 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_d4530066c45445778606ea4bbc21df36 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 2 got 0
I20260501 13:57:29.877777 1881 tablet_bootstrap.cc:654] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.878146 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_d22010815be1426a92087cb895458676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:29.880039 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d22010815be1426a92087cb895458676 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.878391 1128 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_3fd69d2d471947c68f1becc5206a9c0d" } modify_external_catalogs: true
I20260501 13:57:29.881327 1128 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_3fd69d2d471947c68f1becc5206a9c0d" } modify_external_catalogs: true
I20260501 13:57:29.883975 1883 tablet_bootstrap.cc:654] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:29.886310 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2c2873572de4889a16fda8fe31ad594 consensus state has no leader
I20260501 13:57:29.892046 1898 ts_tablet_manager.cc:1929] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:29.892270 1604 tablet_service.cc:1511] Processing CreateTablet for tablet c33a1f423bdb44f5843daffe8f73e74a (DEFAULT_TABLE table=default.Table_c6e79351ac8041739c99caa55b4af0ff [id=d98d5437c5bc4e44bcfe27983dfa36e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.892627 1898 log.cc:1199] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/14b221e0725647df9789c2137ababefe
I20260501 13:57:29.893458 1898 ts_tablet_manager.cc:1950] T 14b221e0725647df9789c2137ababefe P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:29.893858 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c33a1f423bdb44f5843daffe8f73e74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.895922 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 14b221e0725647df9789c2137ababefe (table default.Table_fe2abe8bde8846438f0be465b6c5cc2d [id=2d510b353dcc40938a2a8c3e34c99b76]) successfully deleted
W20260501 13:57:29.896845 1117 catalog_manager.cc:4969] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet d74cf3b82ecf4379a81a1e2ec39213c9 because the tablet was not found. No further retry: Not found: Tablet not found: d74cf3b82ecf4379a81a1e2ec39213c9
I20260501 13:57:29.901228 1881 tablet_bootstrap.cc:492] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:29.901842 1881 ts_tablet_manager.cc:1403] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.006s
I20260501 13:57:29.902415 1976 raft_consensus.cc:493] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:29.902973 1976 raft_consensus.cc:515] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.904299 1881 raft_consensus.cc:359] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.904996 1881 raft_consensus.cc:385] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.905011 1976 leader_election.cc:290] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:29.905259 1881 raft_consensus.cc:740] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.905831 1881 consensus_queue.cc:260] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.906661 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45907da8b56f4fe887253c77ae12f9a3" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:29.908272 1719 leader_election.cc:343] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:29.905750 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45907da8b56f4fe887253c77ae12f9a3" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:29.910135 1717 leader_election.cc:343] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:29.910530 1717 leader_election.cc:304] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:29.913409 1760 tablet_service.cc:1511] Processing CreateTablet for tablet c33a1f423bdb44f5843daffe8f73e74a (DEFAULT_TABLE table=default.Table_c6e79351ac8041739c99caa55b4af0ff [id=d98d5437c5bc4e44bcfe27983dfa36e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.914708 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c33a1f423bdb44f5843daffe8f73e74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.919489 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 215a6200189c40d6aaebd6f1889b88c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:29.919984 1881 ts_tablet_manager.cc:1434] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260501 13:57:29.920473 1882 tablet_bootstrap.cc:492] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:29.920972 1882 ts_tablet_manager.cc:1403] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.404s user 0.015s sys 0.003s
I20260501 13:57:29.921322 1129 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_f2b43b1500494a4bac22cfa9b776b136 [id=0dc2b91768f84b4d93b18f304d31883e]). Expected version 1 got 0
I20260501 13:57:29.922681 1976 raft_consensus.cc:2749] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:29.923543 1881 tablet_bootstrap.cc:492] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:29.923756 1898 tablet_replica.cc:333] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:29.924417 1898 raft_consensus.cc:2243] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:29.924825 1898 raft_consensus.cc:2272] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:29.929176 1881 tablet_bootstrap.cc:654] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:29.941888 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 104708eb7fc0412dad0bd7e14eee9d13 (DEFAULT_TABLE table=default.Table_4a57dcc1c3194af793a3a0367e766cc6 [id=a921f986b8ac4a948ccb297e1084fccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.943002 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104708eb7fc0412dad0bd7e14eee9d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.953292 1899 ts_tablet_manager.cc:1929] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:29.953718 1899 log.cc:1199] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/d74cf3b82ecf4379a81a1e2ec39213c9
I20260501 13:57:29.954959 1899 ts_tablet_manager.cc:1950] T d74cf3b82ecf4379a81a1e2ec39213c9 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:29.955858 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 104708eb7fc0412dad0bd7e14eee9d13 (DEFAULT_TABLE table=default.Table_4a57dcc1c3194af793a3a0367e766cc6 [id=a921f986b8ac4a948ccb297e1084fccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.956921 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104708eb7fc0412dad0bd7e14eee9d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.962518 1882 raft_consensus.cc:359] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.963901 1882 raft_consensus.cc:385] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.964309 1882 raft_consensus.cc:740] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.937542 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_37f461bb7f1c453fbeeab99b079891fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:29.967653 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37f461bb7f1c453fbeeab99b079891fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.965279 1882 consensus_queue.cc:260] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.938778 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_e5f85f68afd24518b03b05a23700361d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:29.972075 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5f85f68afd24518b03b05a23700361d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.973723 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet d74cf3b82ecf4379a81a1e2ec39213c9 (table default.Table_734dfe3b26744b97baf53fd15e44c637 [id=9bf3fd03858a42a2ba91370283546ecf]) successfully deleted
I20260501 13:57:29.939884 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_41c886e2fe374f75b43ad2a436f9eb2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:29.975778 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41c886e2fe374f75b43ad2a436f9eb2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:29.976347 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 104708eb7fc0412dad0bd7e14eee9d13 (DEFAULT_TABLE table=default.Table_4a57dcc1c3194af793a3a0367e766cc6 [id=a921f986b8ac4a948ccb297e1084fccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.977421 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104708eb7fc0412dad0bd7e14eee9d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.978961 1129 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_d4530066c45445778606ea4bbc21df36 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 2 got 0
I20260501 13:57:29.988381 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 2ee580bb9d3843f9bbeae8eef2c347ad (DEFAULT_TABLE table=default.Table_abb70b9287914fbc97d3dcf78ac373bf [id=fe9f5542710e48e8bbf9a48e07c412d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:29.989657 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ee580bb9d3843f9bbeae8eef2c347ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:29.994089 1883 tablet_bootstrap.cc:492] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:29.994587 1883 ts_tablet_manager.cc:1403] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.189s user 0.006s sys 0.008s
I20260501 13:57:29.996831 1883 raft_consensus.cc:359] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:29.997433 1883 raft_consensus.cc:385] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:29.997686 1883 raft_consensus.cc:740] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:29.998230 1883 consensus_queue.cc:260] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.001212 1883 ts_tablet_manager.cc:1434] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260501 13:57:30.002075 1883 tablet_bootstrap.cc:492] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:29.947147 1132 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_2edf812ced814de8acadfe87bcad5893" } modify_external_catalogs: true
I20260501 13:57:30.005064 1132 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_2edf812ced814de8acadfe87bcad5893" } modify_external_catalogs: true
I20260501 13:57:30.008370 1883 tablet_bootstrap.cc:654] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.011710 1901 raft_consensus.cc:493] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:30.012145 1901 raft_consensus.cc:515] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.015709 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e15d66bf6147a08604daf2a9c53322" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:30.015735 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e15d66bf6147a08604daf2a9c53322" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:30.017203 1719 leader_election.cc:343] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.021229 1898 ts_tablet_manager.cc:1916] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:30.028165 1883 tablet_bootstrap.cc:492] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:30.028586 1883 ts_tablet_manager.cc:1403] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.003s
I20260501 13:57:30.031924 1128 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 9f7934ee1b3c4774862c34afa17c63c1
W20260501 13:57:30.037573 1717 leader_election.cc:343] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.038156 1717 leader_election.cc:304] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:30.038909 1976 raft_consensus.cc:2749] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:30.030961 1883 raft_consensus.cc:359] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.039979 1883 raft_consensus.cc:385] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.040231 1883 raft_consensus.cc:740] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.040098 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_77b89c72ca0c4f278237ee79c17b6693" } new_table_name: "default.Table_0f92e0bd075444299cec47f754c2388d" modify_external_catalogs: true
I20260501 13:57:30.043503 1901 leader_election.cc:290] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:30.045625 1883 consensus_queue.cc:260] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.048890 1883 ts_tablet_manager.cc:1434] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260501 13:57:30.049695 1883 tablet_bootstrap.cc:492] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:30.062391 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_f2b43b1500494a4bac22cfa9b776b136 [id=0dc2b91768f84b4d93b18f304d31883e]). Expected version 1 got 0
I20260501 13:57:30.063156 1128 catalog_manager.cc:5671] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: HEALTHY } } }
I20260501 13:57:30.065598 1882 ts_tablet_manager.cc:1434] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.144s user 0.003s sys 0.004s
I20260501 13:57:30.070703 1883 tablet_bootstrap.cc:654] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.072487 1931 raft_consensus.cc:493] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:30.072942 1931 raft_consensus.cc:515] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.074707 1931 leader_election.cc:290] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:30.075325 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 2ee580bb9d3843f9bbeae8eef2c347ad (DEFAULT_TABLE table=default.Table_abb70b9287914fbc97d3dcf78ac373bf [id=fe9f5542710e48e8bbf9a48e07c412d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.076236 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c2873572de4889a16fda8fe31ad594" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:30.076574 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c2873572de4889a16fda8fe31ad594" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:30.076937 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ee580bb9d3843f9bbeae8eef2c347ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.076879 1783 raft_consensus.cc:2468] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:30.077222 1882 tablet_bootstrap.cc:492] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:30.077113 1630 raft_consensus.cc:2468] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:30.078264 1421 leader_election.cc:304] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:30.079075 1931 raft_consensus.cc:2804] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:30.079430 1931 raft_consensus.cc:493] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:30.079746 1931 raft_consensus.cc:3060] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.086000 1931 raft_consensus.cc:515] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.089331 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c2873572de4889a16fda8fe31ad594" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:30.089917 1630 raft_consensus.cc:3060] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.090186 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_d4530066c45445778606ea4bbc21df36" } new_table_name: "default.Table_da800013eb694abeba00d698e4bbc439" modify_external_catalogs: true
I20260501 13:57:30.094733 1989 tablet.cc:1724] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:30.097971 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2c2873572de4889a16fda8fe31ad594" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:30.098503 1783 raft_consensus.cc:3060] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.105130 1783 raft_consensus.cc:2468] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:30.106166 1418 leader_election.cc:304] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:30.107100 1969 raft_consensus.cc:2804] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:30.109261 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 2ee580bb9d3843f9bbeae8eef2c347ad (DEFAULT_TABLE table=default.Table_abb70b9287914fbc97d3dcf78ac373bf [id=fe9f5542710e48e8bbf9a48e07c412d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.110217 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ee580bb9d3843f9bbeae8eef2c347ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.115680 1931 leader_election.cc:290] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:30.116403 1132 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet e41b43329e054442bc8697cc28161954
I20260501 13:57:30.118111 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet e41b43329e054442bc8697cc28161954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:30.122828 1969 raft_consensus.cc:697] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:30.124253 1969 consensus_queue.cc:237] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.129482 1881 tablet_bootstrap.cc:492] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:30.096349 1630 raft_consensus.cc:2468] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:30.131515 1881 ts_tablet_manager.cc:1403] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.208s user 0.012s sys 0.033s
I20260501 13:57:30.132175 1760 tablet_service.cc:1558] Processing DeleteTablet for tablet e41b43329e054442bc8697cc28161954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:30.134692 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet e41b43329e054442bc8697cc28161954 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e41b43329e054442bc8697cc28161954 already in progress: creating tablet
I20260501 13:57:30.136543 1881 raft_consensus.cc:359] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.136853 1603 tablet_service.cc:1558] Processing DeleteTablet for tablet 215a6200189c40d6aaebd6f1889b88c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:30.137548 1897 tablet_replica.cc:333] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:30.138232 1897 raft_consensus.cc:2243] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:30.138689 1897 raft_consensus.cc:2272] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:30.140677 1897 ts_tablet_manager.cc:1916] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:30.161834 1127 catalog_manager.cc:5671] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "993b9ba18d8547f8af43167e727a6a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:30.163441 1127 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet d2c2873572de4889a16fda8fe31ad594 (table default.Table_3d853a6bfdc945fd9fb7c23468905983 [id=b8901f06222746b8bf418391866b92cc]). Expected version 2 got 0
I20260501 13:57:30.137208 1881 raft_consensus.cc:385] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.171896 1881 raft_consensus.cc:740] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
W20260501 13:57:30.172216 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 920d7f9748ae403d90573d612feeba59 has no consensus state
I20260501 13:57:30.173851 1881 consensus_queue.cc:260] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.177577 1881 ts_tablet_manager.cc:1434] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20260501 13:57:30.169960 1898 ts_tablet_manager.cc:1929] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:30.178567 1898 log.cc:1199] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/215a6200189c40d6aaebd6f1889b88c1
I20260501 13:57:30.179247 1898 ts_tablet_manager.cc:1950] T 215a6200189c40d6aaebd6f1889b88c1 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:30.182538 1883 tablet_bootstrap.cc:492] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:30.182924 1883 ts_tablet_manager.cc:1403] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.133s user 0.002s sys 0.009s
I20260501 13:57:30.185392 1883 raft_consensus.cc:359] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.185987 1883 raft_consensus.cc:385] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.186213 1883 raft_consensus.cc:740] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.187752 1897 ts_tablet_manager.cc:1929] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:30.188107 1897 log.cc:1199] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/215a6200189c40d6aaebd6f1889b88c1
I20260501 13:57:30.188694 1897 ts_tablet_manager.cc:1950] T 215a6200189c40d6aaebd6f1889b88c1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:30.191774 1881 tablet_bootstrap.cc:492] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:30.196887 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 215a6200189c40d6aaebd6f1889b88c1 (table default.Table_818fd205f01f45deaaf30421db6a783d [id=59162f26c9244b389d0fda77569be02a]) successfully deleted
I20260501 13:57:30.196530 1883 consensus_queue.cc:260] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:30.197857 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet e41b43329e054442bc8697cc28161954 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e41b43329e054442bc8697cc28161954 already in progress: creating tablet
I20260501 13:57:30.204581 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_a74b32dea54d43a2a8a1b6f2210f38d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:30.206064 1969 raft_consensus.cc:493] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:30.206521 1969 raft_consensus.cc:515] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:30.206288 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a74b32dea54d43a2a8a1b6f2210f38d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:30.209277 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf342e482ca04727a10cf79519549b69" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:30.210491 1421 leader_election.cc:343] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.212729 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf342e482ca04727a10cf79519549b69" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:30.213330 1783 raft_consensus.cc:2468] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:30.214874 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet e41b43329e054442bc8697cc28161954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:30.215019 1418 leader_election.cc:304] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:30.215937 1990 raft_consensus.cc:2804] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:30.216261 1990 raft_consensus.cc:493] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:30.216570 1990 raft_consensus.cc:3060] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.216964 1882 tablet_bootstrap.cc:654] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.224627 1969 leader_election.cc:290] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:30.228077 1990 raft_consensus.cc:515] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.231668 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf342e482ca04727a10cf79519549b69" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:30.233510 1131 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 3 got 2
I20260501 13:57:30.235139 1990 leader_election.cc:290] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:30.235831 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf342e482ca04727a10cf79519549b69" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:30.236408 1782 raft_consensus.cc:3060] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.238639 1883 ts_tablet_manager.cc:1434] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.055s user 0.006s sys 0.000s
I20260501 13:57:30.239557 1883 tablet_bootstrap.cc:492] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:30.240020 1129 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_abb70b9287914fbc97d3dcf78ac373bf" } modify_external_catalogs: true
I20260501 13:57:30.240550 1129 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_abb70b9287914fbc97d3dcf78ac373bf" } modify_external_catalogs: true
I20260501 13:57:30.243873 1995 tablet.cc:1724] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260501 13:57:30.244957 1421 leader_election.cc:343] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.246021 1881 tablet_bootstrap.cc:654] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.252835 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet f2b59b51d81249e0ad90edba16803285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:30.254402 1899 tablet_replica.cc:333] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:30.255199 1899 raft_consensus.cc:2243] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:30.255614 1899 raft_consensus.cc:2272] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:30.257526 1899 ts_tablet_manager.cc:1916] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:30.262499 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 215a6200189c40d6aaebd6f1889b88c1 (table default.Table_818fd205f01f45deaaf30421db6a783d [id=59162f26c9244b389d0fda77569be02a]) successfully deleted
I20260501 13:57:30.261006 1883 tablet_bootstrap.cc:654] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:30.273447 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet e41b43329e054442bc8697cc28161954 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e41b43329e054442bc8697cc28161954 already in progress: creating tablet
I20260501 13:57:30.273797 1883 tablet_bootstrap.cc:492] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:30.274201 1883 ts_tablet_manager.cc:1403] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.000s
I20260501 13:57:30.275915 1881 tablet_bootstrap.cc:492] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:30.276311 1881 ts_tablet_manager.cc:1403] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.002s
I20260501 13:57:30.278596 1881 raft_consensus.cc:359] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.279227 1881 raft_consensus.cc:385] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.279484 1881 raft_consensus.cc:740] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.282725 1881 consensus_queue.cc:260] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.285485 1883 raft_consensus.cc:359] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.286113 1883 raft_consensus.cc:385] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.286353 1883 raft_consensus.cc:740] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.286769 1782 raft_consensus.cc:2468] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:30.287807 1418 leader_election.cc:304] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:30.294452 1881 ts_tablet_manager.cc:1434] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260501 13:57:30.296290 1882 tablet_bootstrap.cc:492] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:30.296694 1882 ts_tablet_manager.cc:1403] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.220s user 0.008s sys 0.004s
I20260501 13:57:30.297065 1881 tablet_bootstrap.cc:492] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:30.298964 1899 ts_tablet_manager.cc:1929] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:30.302289 1899 log.cc:1199] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/f2b59b51d81249e0ad90edba16803285
I20260501 13:57:30.302177 1882 raft_consensus.cc:359] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.286953 1883 consensus_queue.cc:260] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.306423 1899 ts_tablet_manager.cc:1950] T f2b59b51d81249e0ad90edba16803285 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:30.306160 1882 raft_consensus.cc:385] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.307646 1882 raft_consensus.cc:740] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.310204 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet f2b59b51d81249e0ad90edba16803285 (table default.Table_7889b062d4464b39a2ce4ad2eef30dee [id=0444fad8c66840f2a003291fa6cc7208]) successfully deleted
I20260501 13:57:30.319794 1125 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 3 got 2
I20260501 13:57:30.311947 1882 consensus_queue.cc:260] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.321245 1883 ts_tablet_manager.cc:1434] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.047s user 0.007s sys 0.000s
I20260501 13:57:30.322142 1883 tablet_bootstrap.cc:492] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:30.324721 1132 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_9e3d29db155f4fd2ba6229fc2318edb0" } modify_external_catalogs: true
I20260501 13:57:30.325250 1132 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_9e3d29db155f4fd2ba6229fc2318edb0" } modify_external_catalogs: true
I20260501 13:57:30.328649 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_95c99739e2574437b16501060ce816af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:30.330420 1882 ts_tablet_manager.cc:1434] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.033s user 0.004s sys 0.004s
W20260501 13:57:30.330299 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95c99739e2574437b16501060ce816af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:30.339856 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_6be94aa0ac634ce19d10a1c97e33fd5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:30.341166 1882 tablet_bootstrap.cc:492] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:30.343212 1881 tablet_bootstrap.cc:654] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.346165 1990 raft_consensus.cc:2804] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:30.346776 1990 raft_consensus.cc:697] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:30.347651 1990 consensus_queue.cc:237] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:30.347185 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6be94aa0ac634ce19d10a1c97e33fd5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:30.351189 1969 raft_consensus.cc:493] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:30.351440 1995 tablet.cc:1719] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: Skipping requested alter to schema version 2, tablet already version 2
I20260501 13:57:30.351678 1969 raft_consensus.cc:515] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.353150 1984 raft_consensus.cc:493] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:30.353693 1969 leader_election.cc:290] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:30.353566 1984 raft_consensus.cc:515] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.354866 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "921e5c3eaaa44c62abbc90f560049614" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:30.355425 1984 leader_election.cc:290] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:30.357429 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "921e5c3eaaa44c62abbc90f560049614" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:30.357702 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf342e482ca04727a10cf79519549b69" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:30.361213 1989 tablet.cc:1719] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: Skipping requested alter to schema version 2, tablet already version 2
I20260501 13:57:30.361786 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_f8e9d88347b54e2da903776319f1c181"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:30.363446 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8e9d88347b54e2da903776319f1c181 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:30.364873 1418 leader_election.cc:304] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:30.371358 1883 tablet_bootstrap.cc:654] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.374305 1995 tablet.cc:1724] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260501 13:57:30.374617 1969 raft_consensus.cc:2749] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:30.378104 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_bda4446a8e3c4d5ab3d69dc4aeab8b4f" } new_table_name: "default.Table_299166301fa440eca9a509c07bde6ab3" modify_external_catalogs: true
I20260501 13:57:30.380580 1881 tablet_bootstrap.cc:492] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:30.382920 1881 ts_tablet_manager.cc:1403] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.086s user 0.011s sys 0.000s
I20260501 13:57:30.384346 1989 tablet.cc:1724] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260501 13:57:30.382819 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf342e482ca04727a10cf79519549b69" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:30.385569 1881 raft_consensus.cc:359] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.386152 1881 raft_consensus.cc:385] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.386234 1782 raft_consensus.cc:2393] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f90b7ff170b4eadb27abc9ef866ffcf in current term 1: Already voted for candidate 993b9ba18d8547f8af43167e727a6a24 in this term.
I20260501 13:57:30.387266 1563 leader_election.cc:304] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:30.388381 1984 raft_consensus.cc:3060] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.393961 1883 tablet_bootstrap.cc:492] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:30.397547 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_57db8a12223e46c38099f51c4d3d370e" } new_table_name: "default.Table_c4edbd4ba4c54b1e9f9909c9badd7c54" modify_external_catalogs: true
I20260501 13:57:30.400805 1984 raft_consensus.cc:2749] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:30.394367 1883 ts_tablet_manager.cc:1403] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.000s
I20260501 13:57:30.386399 1881 raft_consensus.cc:740] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.406644 1881 consensus_queue.cc:260] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.416190 1901 raft_consensus.cc:493] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:30.416815 1901 raft_consensus.cc:515] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.418593 1901 leader_election.cc:290] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:30.419498 1881 ts_tablet_manager.cc:1434] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20260501 13:57:30.404767 1883 raft_consensus.cc:359] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.420121 1883 raft_consensus.cc:385] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.420150 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5d4064154c744198fa438cd3e3b7288" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:30.420374 1883 raft_consensus.cc:740] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.420946 1883 consensus_queue.cc:260] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:30.421542 1719 leader_election.cc:343] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.423164 1485 raft_consensus.cc:3060] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.423317 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5d4064154c744198fa438cd3e3b7288" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:30.424002 1484 raft_consensus.cc:2468] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 0.
W20260501 13:57:30.437203 1126 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e86d6c935d6444b8a953e7be1bdfa51 consensus state has no leader
W20260501 13:57:30.438799 1128 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet adc47b3b2b084d33a12bd28cfad27382 consensus state has no leader
I20260501 13:57:30.440178 1881 tablet_bootstrap.cc:492] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:30.422799 1883 ts_tablet_manager.cc:1434] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260501 13:57:30.443440 1883 tablet_bootstrap.cc:492] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:30.425659 1717 leader_election.cc:304] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:30.446943 1901 raft_consensus.cc:2804] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:30.447280 1901 raft_consensus.cc:493] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:30.447587 1901 raft_consensus.cc:3060] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.455574 1901 raft_consensus.cc:515] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.457090 1881 tablet_bootstrap.cc:654] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.457620 1901 leader_election.cc:290] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:30.458122 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5d4064154c744198fa438cd3e3b7288" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:30.458246 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5d4064154c744198fa438cd3e3b7288" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:30.458629 1484 raft_consensus.cc:3060] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:30.459574 1719 leader_election.cc:343] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.432026 1882 tablet_bootstrap.cc:654] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.469080 1484 raft_consensus.cc:2468] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 1.
I20260501 13:57:30.470194 1717 leader_election.cc:304] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:30.471243 1901 raft_consensus.cc:2804] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:30.471912 1901 raft_consensus.cc:697] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Running, Role: LEADER
I20260501 13:57:30.472146 1990 raft_consensus.cc:493] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:30.472514 1883 tablet_bootstrap.cc:654] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.472532 1990 raft_consensus.cc:515] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.472851 1901 consensus_queue.cc:237] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.477488 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8d0f6d1f38b428291dba22015ac6959" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:30.478672 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8d0f6d1f38b428291dba22015ac6959" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:30.479250 1782 raft_consensus.cc:2468] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:30.480551 1418 leader_election.cc:304] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:30.481343 1969 raft_consensus.cc:2804] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:30.481637 1969 raft_consensus.cc:493] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:30.481921 1969 raft_consensus.cc:3060] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.485611 1976 raft_consensus.cc:493] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:30.486034 1976 raft_consensus.cc:515] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.487838 1976 leader_election.cc:290] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
W20260501 13:57:30.489127 1421 leader_election.cc:343] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.491165 1883 tablet_bootstrap.cc:492] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:30.491179 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf96cb4696704478a2064d6ce785d49d" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:30.491564 1883 ts_tablet_manager.cc:1403] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.001s
W20260501 13:57:30.492388 1719 leader_election.cc:343] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.474295 1990 leader_election.cc:290] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:30.493889 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf96cb4696704478a2064d6ce785d49d" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:30.495103 1717 leader_election.cc:343] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.495493 1717 leader_election.cc:304] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:30.495844 1881 tablet_bootstrap.cc:492] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:30.496213 1881 ts_tablet_manager.cc:1403] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.000s
I20260501 13:57:30.497368 1969 raft_consensus.cc:515] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.499447 1881 raft_consensus.cc:359] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.500038 1881 raft_consensus.cc:385] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.499995 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8d0f6d1f38b428291dba22015ac6959" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:30.500267 1881 raft_consensus.cc:740] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.500780 1881 consensus_queue.cc:260] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:30.501519 1421 leader_election.cc:343] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.504729 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8d0f6d1f38b428291dba22015ac6959" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:30.505288 1782 raft_consensus.cc:3060] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.511566 1976 raft_consensus.cc:2749] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:30.511618 1883 raft_consensus.cc:359] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.512312 1883 raft_consensus.cc:385] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.512557 1883 raft_consensus.cc:740] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.513100 1883 consensus_queue.cc:260] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.515035 1782 raft_consensus.cc:2468] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:30.516119 1418 leader_election.cc:304] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:30.516858 1881 ts_tablet_manager.cc:1434] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260501 13:57:30.517243 1990 raft_consensus.cc:2804] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:30.517692 1990 raft_consensus.cc:697] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:30.519593 1881 tablet_bootstrap.cc:492] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:30.518978 1990 consensus_queue.cc:237] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.524063 1969 leader_election.cc:290] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:30.529320 1883 ts_tablet_manager.cc:1434] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20260501 13:57:30.530731 1883 tablet_bootstrap.cc:492] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:30.532502 1881 tablet_bootstrap.cc:654] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:30.539644 2003 log.cc:927] Time spent T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260501 13:57:30.543557 1881 tablet_bootstrap.cc:492] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:30.543973 1881 ts_tablet_manager.cc:1403] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.000s
I20260501 13:57:30.546375 1881 raft_consensus.cc:359] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.547008 1881 raft_consensus.cc:385] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.547367 1881 raft_consensus.cc:740] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.547976 1881 consensus_queue.cc:260] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.548032 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_c7461471f08c462a93d877fb770f1ab9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:30.549562 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7461471f08c462a93d877fb770f1ab9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:30.549751 1881 ts_tablet_manager.cc:1434] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260501 13:57:30.550624 1881 tablet_bootstrap.cc:492] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:30.559372 1881 tablet_bootstrap.cc:654] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.568246 1969 raft_consensus.cc:493] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:30.568754 1969 raft_consensus.cc:515] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.571432 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45907da8b56f4fe887253c77ae12f9a3" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:30.572877 1421 leader_election.cc:343] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.575827 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_6452fc6a9103447c9239a13e344f0c5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:30.577250 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6452fc6a9103447c9239a13e344f0c5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:30.585696 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45907da8b56f4fe887253c77ae12f9a3" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:30.586380 1782 raft_consensus.cc:2468] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:30.587450 1418 leader_election.cc:304] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:30.588388 1990 raft_consensus.cc:2804] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:30.588761 1990 raft_consensus.cc:493] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:30.589140 1990 raft_consensus.cc:3060] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.594246 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_8999f52c517a462a9845800aba73657e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:30.595896 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8999f52c517a462a9845800aba73657e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:30.598599 1969 leader_election.cc:290] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:30.566177 2004 tablet.cc:1724] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:30.615216 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_183071821cf844028ee50b8498ff77a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:30.617206 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_183071821cf844028ee50b8498ff77a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:30.598404 1990 raft_consensus.cc:515] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.622646 1131 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 89e2c22052d04fe1a21bcb3323385111 (table default.Table_4b9a916d6a754ac7b38819b39d0b1671 [id=36e8cb1c80cc45e7b4c501608c374af9]). Expected version 1 got 0
I20260501 13:57:30.623183 1131 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 3 got 2
I20260501 13:57:30.624433 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 1409fbe5846a4bb099f31fb2d2e41382 (DEFAULT_TABLE table=default.Table_d22010815be1426a92087cb895458676 [id=8d39f1f13bc24a2c84e888bb29b6e74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.625483 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1409fbe5846a4bb099f31fb2d2e41382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.632233 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 1409fbe5846a4bb099f31fb2d2e41382 (DEFAULT_TABLE table=default.Table_d22010815be1426a92087cb895458676 [id=8d39f1f13bc24a2c84e888bb29b6e74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.633275 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1409fbe5846a4bb099f31fb2d2e41382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.635177 1760 tablet_service.cc:1511] Processing CreateTablet for tablet b507a4d4c9c54bc68d92d9286794a264 (DEFAULT_TABLE table=default.Table_2294cbb031954b92a89e152ac50cb9f5 [id=36cedf51b8a8402eb9488172828f7eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.635979 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 1409fbe5846a4bb099f31fb2d2e41382 (DEFAULT_TABLE table=default.Table_d22010815be1426a92087cb895458676 [id=8d39f1f13bc24a2c84e888bb29b6e74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.636732 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b507a4d4c9c54bc68d92d9286794a264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.637049 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1409fbe5846a4bb099f31fb2d2e41382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.638176 1458 tablet_service.cc:1511] Processing CreateTablet for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 (DEFAULT_TABLE table=default.Table_4120cd7643bd4ca8bdd6b6c163c8e375 [id=ae0dad3b628b454d899e4912c8013d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.638720 1459 tablet_service.cc:1511] Processing CreateTablet for tablet b507a4d4c9c54bc68d92d9286794a264 (DEFAULT_TABLE table=default.Table_2294cbb031954b92a89e152ac50cb9f5 [id=36cedf51b8a8402eb9488172828f7eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.639256 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.640045 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b507a4d4c9c54bc68d92d9286794a264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.639864 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 920d7f9748ae403d90573d612feeba59 (DEFAULT_TABLE table=default.Table_0f92e0bd075444299cec47f754c2388d [id=068f5ac17d494d0b8639cbdc07a04597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.640858 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920d7f9748ae403d90573d612feeba59. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:30.644285 1119 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 5e86d6c935d6444b8a953e7be1bdfa51 (table default.Table_299166301fa440eca9a509c07bde6ab3 [id=0f9c1230865848adaa4e5ea744c31c8b], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5e86d6c935d6444b8a953e7be1bdfa51 consensus state has no leader
I20260501 13:57:30.649788 1604 tablet_service.cc:1511] Processing CreateTablet for tablet b507a4d4c9c54bc68d92d9286794a264 (DEFAULT_TABLE table=default.Table_2294cbb031954b92a89e152ac50cb9f5 [id=36cedf51b8a8402eb9488172828f7eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.651131 1603 tablet_service.cc:1511] Processing CreateTablet for tablet 920d7f9748ae403d90573d612feeba59 (DEFAULT_TABLE table=default.Table_0f92e0bd075444299cec47f754c2388d [id=068f5ac17d494d0b8639cbdc07a04597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.652252 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 (DEFAULT_TABLE table=default.Table_4120cd7643bd4ca8bdd6b6c163c8e375 [id=ae0dad3b628b454d899e4912c8013d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.658025 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b507a4d4c9c54bc68d92d9286794a264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.658828 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45907da8b56f4fe887253c77ae12f9a3" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:30.660095 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920d7f9748ae403d90573d612feeba59. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:30.660032 1421 leader_election.cc:343] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.662998 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.674767 1630 raft_consensus.cc:1275] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Refusing update from remote peer 993b9ba18d8547f8af43167e727a6a24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:57:30.678129 1969 consensus_queue.cc:1048] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260501 13:57:30.655273 1883 tablet_bootstrap.cc:654] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.656432 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 (DEFAULT_TABLE table=default.Table_4120cd7643bd4ca8bdd6b6c163c8e375 [id=ae0dad3b628b454d899e4912c8013d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.691586 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_0f92e0bd075444299cec47f754c2388d" } new_table_name: "default.Table_f0801819716e426f92844203c4a06bdc" modify_external_catalogs: true
I20260501 13:57:30.692507 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.694445 1882 tablet_bootstrap.cc:492] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:30.694876 1882 ts_tablet_manager.cc:1403] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.354s user 0.011s sys 0.000s
I20260501 13:57:30.666028 1976 raft_consensus.cc:493] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:30.697710 1882 raft_consensus.cc:359] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.697911 1976 raft_consensus.cc:515] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.699630 1976 leader_election.cc:290] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:30.673986 1783 raft_consensus.cc:1275] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Refusing update from remote peer 993b9ba18d8547f8af43167e727a6a24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:57:30.700610 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89e2c22052d04fe1a21bcb3323385111" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:30.700858 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_2294cbb031954b92a89e152ac50cb9f5" } new_table_name: "default.Table_83779be1d30c4839999b9217ce2e4a20" modify_external_catalogs: true
I20260501 13:57:30.655791 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 920d7f9748ae403d90573d612feeba59 (DEFAULT_TABLE table=default.Table_0f92e0bd075444299cec47f754c2388d [id=068f5ac17d494d0b8639cbdc07a04597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:30.702603 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89e2c22052d04fe1a21bcb3323385111" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:30.702955 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920d7f9748ae403d90573d612feeba59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:30.703178 1485 raft_consensus.cc:2468] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 0.
I20260501 13:57:30.705117 1717 leader_election.cc:304] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:30.663164 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45907da8b56f4fe887253c77ae12f9a3" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:30.706418 1782 raft_consensus.cc:3060] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.706638 1976 raft_consensus.cc:2804] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:30.706983 1976 raft_consensus.cc:493] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:30.707269 1976 raft_consensus.cc:3060] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.708041 1969 consensus_queue.cc:1048] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:30.715312 1719 leader_election.cc:343] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.698297 1882 raft_consensus.cc:385] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.663733 1990 leader_election.cc:290] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:30.736804 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet c5297eb8aace46078cc3fce289fdd389 (table default.Table_eb039fccfc7d444baf4e6cdddc6be5cd [id=37089fa463db4a608b09504153e2e32a]). Expected version 1 got 0
I20260501 13:57:30.746743 1129 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 2ee580bb9d3843f9bbeae8eef2c347ad
I20260501 13:57:30.751538 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ee580bb9d3843f9bbeae8eef2c347ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:30.745761 1125 catalog_manager.cc:5671] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: HEALTHY } } }
I20260501 13:57:30.758785 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_f2b43b1500494a4bac22cfa9b776b136 [id=0dc2b91768f84b4d93b18f304d31883e]). Expected version 1 got 0
I20260501 13:57:30.764935 2004 tablet.cc:1724] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:30.765630 2008 mvcc.cc:204] Tried to move back new op lower bound from 7281229212321697792 to 7281229210117591040. Current Snapshot: MvccSnapshot[applied={T|T < 7281229212321697792}]
I20260501 13:57:30.775969 1881 tablet_bootstrap.cc:492] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:30.776404 1881 ts_tablet_manager.cc:1403] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.226s user 0.018s sys 0.033s
I20260501 13:57:30.779513 1881 raft_consensus.cc:359] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.780375 1881 raft_consensus.cc:385] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.780709 1881 raft_consensus.cc:740] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.781461 1881 consensus_queue.cc:260] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.788024 1881 ts_tablet_manager.cc:1434] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260501 13:57:30.794314 1881 tablet_bootstrap.cc:492] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:30.800266 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ee580bb9d3843f9bbeae8eef2c347ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:30.800714 1717 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.1.16.189:45897, user_credentials={real_user=slave}} blocked reactor thread for 80388.7us
W20260501 13:57:30.802325 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 2ee580bb9d3843f9bbeae8eef2c347ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ee580bb9d3843f9bbeae8eef2c347ad already in progress: creating tablet
I20260501 13:57:30.798427 2015 tablet.cc:1719] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: Skipping requested alter to schema version 2, tablet already version 2
I20260501 13:57:30.809610 1976 raft_consensus.cc:515] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.813553 2015 tablet.cc:1724] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260501 13:57:30.720772 1882 raft_consensus.cc:740] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.749163 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ee580bb9d3843f9bbeae8eef2c347ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:30.764505 1989 tablet.cc:1724] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:30.843864 1782 raft_consensus.cc:2468] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:30.845075 1418 leader_election.cc:304] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:30.846212 1990 raft_consensus.cc:2804] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:30.846566 1976 leader_election.cc:290] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:30.847219 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89e2c22052d04fe1a21bcb3323385111" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:30.847733 1485 raft_consensus.cc:3060] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:30.849426 1990 raft_consensus.cc:697] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:30.851044 1990 consensus_queue.cc:237] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.852136 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89e2c22052d04fe1a21bcb3323385111" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
W20260501 13:57:30.854944 1719 leader_election.cc:343] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:30.842321 1882 consensus_queue.cc:260] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.869235 1882 ts_tablet_manager.cc:1434] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.174s user 0.005s sys 0.000s
I20260501 13:57:30.870126 1882 tablet_bootstrap.cc:492] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:30.875674 1995 tablet.cc:1724] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:30.870715 1485 raft_consensus.cc:2468] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 1.
I20260501 13:57:30.883569 1132 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 4a563bccc7b046658ed133109f676528
I20260501 13:57:30.886611 1883 tablet_bootstrap.cc:492] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:30.887080 1883 ts_tablet_manager.cc:1403] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.357s user 0.011s sys 0.000s
I20260501 13:57:30.888438 1607 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a563bccc7b046658ed133109f676528 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:30.889583 1883 raft_consensus.cc:359] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.890251 1883 raft_consensus.cc:385] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.895911 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.188:38815
--webserver_interface=127.1.16.188
--webserver_port=41671
--builtin_ntp_servers=127.1.16.148:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260501 13:57:30.897161 1883 raft_consensus.cc:740] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.897831 1883 consensus_queue.cc:260] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.899812 1883 ts_tablet_manager.cc:1434] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260501 13:57:30.909557 1883 tablet_bootstrap.cc:492] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:30.911942 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a563bccc7b046658ed133109f676528 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:30.912369 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b507a4d4c9c54bc68d92d9286794a264 consensus state has no leader
W20260501 13:57:30.916482 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 920d7f9748ae403d90573d612feeba59 consensus state has no leader
I20260501 13:57:30.919662 1883 tablet_bootstrap.cc:654] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:30.919999 1717 leader_election.cc:304] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:30.921044 1976 raft_consensus.cc:2804] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:30.921453 1976 raft_consensus.cc:697] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Running, Role: LEADER
I20260501 13:57:30.922168 1976 consensus_queue.cc:237] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.926703 1131 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_f2b43b1500494a4bac22cfa9b776b136 [id=0dc2b91768f84b4d93b18f304d31883e]). Expected version 1 got 0
I20260501 13:57:30.927379 1131 catalog_manager.cc:5671] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "993b9ba18d8547f8af43167e727a6a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
W20260501 13:57:30.928681 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 4a563bccc7b046658ed133109f676528 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a563bccc7b046658ed133109f676528 already in progress: creating tablet
I20260501 13:57:30.928768 1131 catalog_manager.cc:5671] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "993b9ba18d8547f8af43167e727a6a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:30.930064 1131 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47]). Expected version 3 got 2
I20260501 13:57:30.932193 1131 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 45907da8b56f4fe887253c77ae12f9a3 (table default.Table_4275f119eec547c28a46b8d5f4faa0f8 [id=5cd8cae1c55f41aa809960d1cecddec3]). Expected version 1 got 0
I20260501 13:57:30.932804 1131 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 89e2c22052d04fe1a21bcb3323385111 (table default.Table_4b9a916d6a754ac7b38819b39d0b1671 [id=36e8cb1c80cc45e7b4c501608c374af9]). Expected version 1 got 0
I20260501 13:57:30.933518 1123 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47] alter complete (version 3)
I20260501 13:57:30.938027 1881 tablet_bootstrap.cc:654] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:30.942785 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 2ee580bb9d3843f9bbeae8eef2c347ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ee580bb9d3843f9bbeae8eef2c347ad already in progress: creating tablet
I20260501 13:57:30.946278 1883 tablet_bootstrap.cc:492] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:30.946768 1883 ts_tablet_manager.cc:1403] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.000s
I20260501 13:57:30.947793 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a563bccc7b046658ed133109f676528 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:30.948057 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet bf96cb4696704478a2064d6ce785d49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:30.948650 1899 tablet_replica.cc:333] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:30.949298 1899 raft_consensus.cc:2243] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:30.949749 1899 raft_consensus.cc:2272] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:30.949958 1883 raft_consensus.cc:359] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.951514 1899 ts_tablet_manager.cc:1916] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:30.952785 1883 raft_consensus.cc:385] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:30.953044 1883 raft_consensus.cc:740] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:30.953606 1883 consensus_queue.cc:260] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:30.955623 1883 ts_tablet_manager.cc:1434] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260501 13:57:30.956568 1883 tablet_bootstrap.cc:492] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:30.970217 1883 tablet_bootstrap.cc:654] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:30.989131 1719 consensus_peers.cc:597] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:30.997980 1883 tablet_bootstrap.cc:492] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:30.998592 1883 ts_tablet_manager.cc:1403] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.042s user 0.023s sys 0.000s
I20260501 13:57:31.001072 1883 raft_consensus.cc:359] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.001804 1883 raft_consensus.cc:385] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:31.002110 1883 raft_consensus.cc:740] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:31.002734 1883 consensus_queue.cc:260] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.005721 1883 ts_tablet_manager.cc:1434] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260501 13:57:31.008034 1883 tablet_bootstrap.cc:492] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:31.008617 1901 consensus_queue.cc:1048] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:31.012487 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 4a563bccc7b046658ed133109f676528 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a563bccc7b046658ed133109f676528 already in progress: creating tablet
I20260501 13:57:31.013254 1883 tablet_bootstrap.cc:654] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:31.020215 1882 tablet_bootstrap.cc:654] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:31.024035 1762 tablet_service.cc:1511] Processing CreateTablet for tablet eeae4369be954d9d9bad846484afb6ff (DEFAULT_TABLE table=default.Table_6452fc6a9103447c9239a13e344f0c5a [id=ecd108cb5a5544c590e904bc9185a0d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.025095 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeae4369be954d9d9bad846484afb6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.028561 1465 tablet_service.cc:1511] Processing CreateTablet for tablet eeae4369be954d9d9bad846484afb6ff (DEFAULT_TABLE table=default.Table_6452fc6a9103447c9239a13e344f0c5a [id=ecd108cb5a5544c590e904bc9185a0d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.028640 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 90a57bc58e854e469897a04b2d434d06 (DEFAULT_TABLE table=default.Table_f8e9d88347b54e2da903776319f1c181 [id=b3e0852bf1bc4eedad8b69f7e4cf0785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.029603 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeae4369be954d9d9bad846484afb6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.029690 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90a57bc58e854e469897a04b2d434d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.029826 1458 tablet_service.cc:1511] Processing CreateTablet for tablet 90a57bc58e854e469897a04b2d434d06 (DEFAULT_TABLE table=default.Table_f8e9d88347b54e2da903776319f1c181 [id=b3e0852bf1bc4eedad8b69f7e4cf0785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.033386 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90a57bc58e854e469897a04b2d434d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.039196 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (DEFAULT_TABLE table=default.Table_6be94aa0ac634ce19d10a1c97e33fd5e [id=1eefeb189133435f960daacad0e615cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.040437 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f9c92846d4e4255a1c8ce3835a0a18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.037912 1131 catalog_manager.cc:5671] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "993b9ba18d8547f8af43167e727a6a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:31.042666 1131 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 45907da8b56f4fe887253c77ae12f9a3 (table default.Table_4275f119eec547c28a46b8d5f4faa0f8 [id=5cd8cae1c55f41aa809960d1cecddec3]). Expected version 1 got 0
I20260501 13:57:31.045698 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (DEFAULT_TABLE table=default.Table_6be94aa0ac634ce19d10a1c97e33fd5e [id=1eefeb189133435f960daacad0e615cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.046831 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f9c92846d4e4255a1c8ce3835a0a18d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:30.994606 1840 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.DeleteTable -> {remote=127.1.16.190:45953, user_credentials={real_user=slave}} blocked reactor thread for 97207.1us
I20260501 13:57:31.056074 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4f8f3a483f7845b0b04a11a999253c87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:31.057736 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f8f3a483f7845b0b04a11a999253c87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:31.069588 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 863a10cfd14c4694b7c6141c0851167a (DEFAULT_TABLE table=default.Table_95c99739e2574437b16501060ce816af [id=a6953bc50bab4e7f92adfee0ef034465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.070679 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863a10cfd14c4694b7c6141c0851167a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.070982 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 90a57bc58e854e469897a04b2d434d06 (DEFAULT_TABLE table=default.Table_f8e9d88347b54e2da903776319f1c181 [id=b3e0852bf1bc4eedad8b69f7e4cf0785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.071869 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90a57bc58e854e469897a04b2d434d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.072130 1607 tablet_service.cc:1511] Processing CreateTablet for tablet eeae4369be954d9d9bad846484afb6ff (DEFAULT_TABLE table=default.Table_6452fc6a9103447c9239a13e344f0c5a [id=ecd108cb5a5544c590e904bc9185a0d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.073036 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeae4369be954d9d9bad846484afb6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.075129 1603 tablet_service.cc:1511] Processing CreateTablet for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (DEFAULT_TABLE table=default.Table_6be94aa0ac634ce19d10a1c97e33fd5e [id=1eefeb189133435f960daacad0e615cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.076081 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f9c92846d4e4255a1c8ce3835a0a18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.098199 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 863a10cfd14c4694b7c6141c0851167a (DEFAULT_TABLE table=default.Table_95c99739e2574437b16501060ce816af [id=a6953bc50bab4e7f92adfee0ef034465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.099246 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863a10cfd14c4694b7c6141c0851167a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.104817 1460 tablet_service.cc:1511] Processing CreateTablet for tablet 9a0d04810f2d465ebe2964797ea30538 (DEFAULT_TABLE table=default.Table_a74b32dea54d43a2a8a1b6f2210f38d8 [id=eaa4c83258f94954af2e04eee604f6f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.105854 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0d04810f2d465ebe2964797ea30538. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:31.116348 1119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.188:38815, user_credentials={real_user=slave}} blocked reactor thread for 46266.3us
W20260501 13:57:31.119818 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 2ee580bb9d3843f9bbeae8eef2c347ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ee580bb9d3843f9bbeae8eef2c347ad already in progress: creating tablet
I20260501 13:57:31.122171 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 9a0d04810f2d465ebe2964797ea30538 (DEFAULT_TABLE table=default.Table_a74b32dea54d43a2a8a1b6f2210f38d8 [id=eaa4c83258f94954af2e04eee604f6f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.123176 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0d04810f2d465ebe2964797ea30538. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:31.127912 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 4a563bccc7b046658ed133109f676528 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a563bccc7b046658ed133109f676528 already in progress: creating tablet
I20260501 13:57:31.130142 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 863a10cfd14c4694b7c6141c0851167a (DEFAULT_TABLE table=default.Table_95c99739e2574437b16501060ce816af [id=a6953bc50bab4e7f92adfee0ef034465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.131242 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 863a10cfd14c4694b7c6141c0851167a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.135246 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 9a0d04810f2d465ebe2964797ea30538 (DEFAULT_TABLE table=default.Table_a74b32dea54d43a2a8a1b6f2210f38d8 [id=eaa4c83258f94954af2e04eee604f6f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.136297 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0d04810f2d465ebe2964797ea30538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.124310 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 8d2f196f44be4fbc8059901380b71271 (table default.Table_4b01fcca8d2a46ccba7050ab708d8a8d [id=8cff6f6a2ebc447c979a45c9bfa438dc]). Expected version 1 got 0
I20260501 13:57:31.140074 1125 catalog_manager.cc:5671] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: HEALTHY } } }
I20260501 13:57:31.141280 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 89e2c22052d04fe1a21bcb3323385111 (table default.Table_4b9a916d6a754ac7b38819b39d0b1671 [id=36e8cb1c80cc45e7b4c501608c374af9]). Expected version 1 got 0
I20260501 13:57:31.144289 1899 ts_tablet_manager.cc:1929] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:31.144635 1899 log.cc:1199] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/bf96cb4696704478a2064d6ce785d49d
I20260501 13:57:31.148486 1976 raft_consensus.cc:493] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:31.148921 1976 raft_consensus.cc:515] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.150032 1602 tablet_service.cc:1511] Processing CreateTablet for tablet 1ba63c153ef04db891cae46a71439455 (DEFAULT_TABLE table=default.Table_37f461bb7f1c453fbeeab99b079891fb [id=e8b19350d702499a82f41a6cce14dbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.150684 1976 leader_election.cc:290] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:31.151072 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba63c153ef04db891cae46a71439455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.152163 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc6b0cfd20174d7680f8981bb850c50f" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:31.153548 1719 leader_election.cc:343] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:31.158529 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 1ba63c153ef04db891cae46a71439455 (DEFAULT_TABLE table=default.Table_37f461bb7f1c453fbeeab99b079891fb [id=e8b19350d702499a82f41a6cce14dbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.159446 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba63c153ef04db891cae46a71439455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.165656 1899 ts_tablet_manager.cc:1950] T bf96cb4696704478a2064d6ce785d49d P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:31.169664 1123 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 45907da8b56f4fe887253c77ae12f9a3 (table default.Table_4275f119eec547c28a46b8d5f4faa0f8 [id=5cd8cae1c55f41aa809960d1cecddec3]). Expected version 1 got 0
I20260501 13:57:31.172128 1131 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_3d853a6bfdc945fd9fb7c23468905983 [id=b8901f06222746b8bf418391866b92cc] alter complete (version 2)
I20260501 13:57:31.185487 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_609f259197a542caaa0dd2fdc258b986"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:31.187342 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_609f259197a542caaa0dd2fdc258b986 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:31.188591 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 6c2d5336bfec43dbb38ce7071834bca3 (table default.Table_66545ca4c3d540ada57273201e17f2d8 [id=697eaa78d5e541988cd48466c64c5915]). Expected version 1 got 0
I20260501 13:57:31.189461 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_af01b0d8a23647a48ebf1ea88cb91b16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:31.191089 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af01b0d8a23647a48ebf1ea88cb91b16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:31.178222 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 6650472ec4a449c48c7957a9bbec88e3 (DEFAULT_TABLE table=default.Table_e5f85f68afd24518b03b05a23700361d [id=774a7eb90ba449b6aef225d819678439]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.192421 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6650472ec4a449c48c7957a9bbec88e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.192989 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 6650472ec4a449c48c7957a9bbec88e3 (DEFAULT_TABLE table=default.Table_e5f85f68afd24518b03b05a23700361d [id=774a7eb90ba449b6aef225d819678439]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.192989 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_bd1dfe371f64485fb2a9b88dca4dd231"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:31.194046 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6650472ec4a449c48c7957a9bbec88e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:31.194504 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd1dfe371f64485fb2a9b88dca4dd231 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:31.196178 1124 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4a57dcc1c3194af793a3a0367e766cc6" } modify_external_catalogs: true
I20260501 13:57:31.190392 1976 raft_consensus.cc:493] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:31.196659 1124 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4a57dcc1c3194af793a3a0367e766cc6" } modify_external_catalogs: true
I20260501 13:57:31.197067 1976 raft_consensus.cc:515] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.199020 1976 leader_election.cc:290] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:31.200284 1763 tablet_service.cc:1511] Processing CreateTablet for tablet b98187c7843e42b09d55c851117c4ac8 (DEFAULT_TABLE table=default.Table_8999f52c517a462a9845800aba73657e [id=e1549f4369a14aceb962e2be6f1a2ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.201457 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98187c7843e42b09d55c851117c4ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.201503 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5297eb8aace46078cc3fce289fdd389" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:31.202950 1719 leader_election.cc:343] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:31.192989 1608 tablet_service.cc:1511] Processing CreateTablet for tablet b98187c7843e42b09d55c851117c4ac8 (DEFAULT_TABLE table=default.Table_8999f52c517a462a9845800aba73657e [id=e1549f4369a14aceb962e2be6f1a2ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.204422 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98187c7843e42b09d55c851117c4ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.205672 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet bf96cb4696704478a2064d6ce785d49d (table default.Table_4c1ccc461d624514a37044a124d8ad6b [id=314708984ce445e2a57e799bc77d5c6f]) successfully deleted
I20260501 13:57:31.218556 1984 raft_consensus.cc:493] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:31.219008 1984 raft_consensus.cc:515] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.220722 1984 leader_election.cc:290] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:31.222081 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45907da8b56f4fe887253c77ae12f9a3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:31.222786 1783 raft_consensus.cc:2393] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f90b7ff170b4eadb27abc9ef866ffcf in current term 1: Already voted for candidate 993b9ba18d8547f8af43167e727a6a24 in this term.
I20260501 13:57:31.224637 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 9018d70c734a429aa59565ad67a46acc (DEFAULT_TABLE table=default.Table_41c886e2fe374f75b43ad2a436f9eb2b [id=454925118d6943458e68bb0089610a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.225570 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9018d70c734a429aa59565ad67a46acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.235114 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45907da8b56f4fe887253c77ae12f9a3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:31.236126 1883 tablet_bootstrap.cc:492] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:31.236513 1883 ts_tablet_manager.cc:1403] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.229s user 0.020s sys 0.016s
I20260501 13:57:31.236996 1610 tablet_service.cc:1511] Processing CreateTablet for tablet 9018d70c734a429aa59565ad67a46acc (DEFAULT_TABLE table=default.Table_41c886e2fe374f75b43ad2a436f9eb2b [id=454925118d6943458e68bb0089610a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.238042 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9018d70c734a429aa59565ad67a46acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.238651 1461 tablet_service.cc:1511] Processing CreateTablet for tablet b98187c7843e42b09d55c851117c4ac8 (DEFAULT_TABLE table=default.Table_8999f52c517a462a9845800aba73657e [id=e1549f4369a14aceb962e2be6f1a2ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.239854 1883 raft_consensus.cc:359] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.240294 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98187c7843e42b09d55c851117c4ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.240463 1883 raft_consensus.cc:385] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:31.240691 1883 raft_consensus.cc:740] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:31.241214 1883 consensus_queue.cc:260] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.243206 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 6650472ec4a449c48c7957a9bbec88e3 (DEFAULT_TABLE table=default.Table_e5f85f68afd24518b03b05a23700361d [id=774a7eb90ba449b6aef225d819678439]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.243443 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 9018d70c734a429aa59565ad67a46acc (DEFAULT_TABLE table=default.Table_41c886e2fe374f75b43ad2a436f9eb2b [id=454925118d6943458e68bb0089610a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.244143 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6650472ec4a449c48c7957a9bbec88e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.244957 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 1ba63c153ef04db891cae46a71439455 (DEFAULT_TABLE table=default.Table_37f461bb7f1c453fbeeab99b079891fb [id=e8b19350d702499a82f41a6cce14dbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.245899 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ba63c153ef04db891cae46a71439455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.248364 1564 leader_election.cc:304] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:31.249464 1984 raft_consensus.cc:3060] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:31.254027 1881 tablet_bootstrap.cc:492] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:31.255123 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc6b0cfd20174d7680f8981bb850c50f" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:31.255367 1881 ts_tablet_manager.cc:1403] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.461s user 0.019s sys 0.022s
I20260501 13:57:31.258085 1984 raft_consensus.cc:2749] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:31.255735 1484 raft_consensus.cc:2468] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 0.
I20260501 13:57:31.261512 1717 leader_election.cc:304] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:31.263906 1976 raft_consensus.cc:2804] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:31.264285 1976 raft_consensus.cc:493] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:31.264567 1976 raft_consensus.cc:3060] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:31.274567 1883 ts_tablet_manager.cc:1434] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20260501 13:57:31.276415 1883 tablet_bootstrap.cc:492] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:31.289266 1605 tablet_service.cc:1511] Processing CreateTablet for tablet ffe89d656956440e8d0c1de0b01ce486 (DEFAULT_TABLE table=default.Table_c7461471f08c462a93d877fb770f1ab9 [id=3c1a635628034d019c34ce2059bee715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.290357 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe89d656956440e8d0c1de0b01ce486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.293619 1762 tablet_service.cc:1511] Processing CreateTablet for tablet ffe89d656956440e8d0c1de0b01ce486 (DEFAULT_TABLE table=default.Table_c7461471f08c462a93d877fb770f1ab9 [id=3c1a635628034d019c34ce2059bee715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.294962 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe89d656956440e8d0c1de0b01ce486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.301643 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9018d70c734a429aa59565ad67a46acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.306022 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 89e2c22052d04fe1a21bcb3323385111 (table default.Table_4b9a916d6a754ac7b38819b39d0b1671 [id=36e8cb1c80cc45e7b4c501608c374af9]). Expected version 1 got 0
I20260501 13:57:31.306157 1883 tablet_bootstrap.cc:654] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:31.306927 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_f2b43b1500494a4bac22cfa9b776b136 [id=0dc2b91768f84b4d93b18f304d31883e]). Expected version 1 got 0
I20260501 13:57:31.309293 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 02926865010043e4b689c914b435400f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:31.309978 1899 tablet_replica.cc:333] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:31.310704 1899 raft_consensus.cc:2243] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:31.311174 1899 raft_consensus.cc:2272] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:31.312911 1899 ts_tablet_manager.cc:1916] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:31.325222 1899 ts_tablet_manager.cc:1929] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:31.282752 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5297eb8aace46078cc3fce289fdd389" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:31.325873 1883 tablet_bootstrap.cc:492] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:31.331111 1883 ts_tablet_manager.cc:1403] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.004s
I20260501 13:57:31.330962 1899 log.cc:1199] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/02926865010043e4b689c914b435400f
I20260501 13:57:31.287905 1990 consensus_queue.cc:1048] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.044s
I20260501 13:57:31.331856 1899 ts_tablet_manager.cc:1950] T 02926865010043e4b689c914b435400f P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:31.334151 1883 raft_consensus.cc:359] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.339660 1883 raft_consensus.cc:385] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:31.339948 1883 raft_consensus.cc:740] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:31.340571 1883 consensus_queue.cc:260] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.352571 1883 ts_tablet_manager.cc:1434] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20260501 13:57:31.353482 1883 tablet_bootstrap.cc:492] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:31.379341 1883 tablet_bootstrap.cc:654] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:31.395140 1883 tablet_bootstrap.cc:492] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:31.397415 1881 raft_consensus.cc:359] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.399536 1463 tablet_service.cc:1511] Processing CreateTablet for tablet ffe89d656956440e8d0c1de0b01ce486 (DEFAULT_TABLE table=default.Table_c7461471f08c462a93d877fb770f1ab9 [id=3c1a635628034d019c34ce2059bee715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.400641 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe89d656956440e8d0c1de0b01ce486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.403059 1883 ts_tablet_manager.cc:1403] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.002s
I20260501 13:57:31.406085 1883 raft_consensus.cc:359] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.415611 1883 raft_consensus.cc:385] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:31.415913 1883 raft_consensus.cc:740] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:31.416515 1883 consensus_queue.cc:260] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.423210 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 02926865010043e4b689c914b435400f (table default.Table_4bb39ebdce2145ccaf94f17bb84197b8 [id=a7c0d6e58eac4f9aabd0f2bd585db9a6]) successfully deleted
I20260501 13:57:31.418316 1883 ts_tablet_manager.cc:1434] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20260501 13:57:31.427172 1883 tablet_bootstrap.cc:492] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:31.437150 1883 tablet_bootstrap.cc:654] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:31.445448 1717 leader_election.cc:343] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:31.446008 1717 leader_election.cc:304] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:31.446798 1901 raft_consensus.cc:2749] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:31.439581 1976 raft_consensus.cc:515] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.451067 1976 leader_election.cc:290] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:31.455890 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc6b0cfd20174d7680f8981bb850c50f" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:31.451617 1901 raft_consensus.cc:493] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:31.458362 1883 tablet_bootstrap.cc:492] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:31.458158 1901 raft_consensus.cc:515] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:31.457612 1719 leader_election.cc:343] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:31.463001 1901 leader_election.cc:290] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:31.458814 1883 ts_tablet_manager.cc:1403] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.004s
I20260501 13:57:31.464103 1882 tablet_bootstrap.cc:492] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:31.464499 1882 ts_tablet_manager.cc:1403] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.595s user 0.037s sys 0.031s
I20260501 13:57:31.464696 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adc47b3b2b084d33a12bd28cfad27382" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:31.466107 1719 leader_election.cc:343] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:31.466611 1883 raft_consensus.cc:359] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.466830 1882 raft_consensus.cc:359] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.467309 1883 raft_consensus.cc:385] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:31.467464 1882 raft_consensus.cc:385] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:31.467602 1883 raft_consensus.cc:740] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:31.467707 1882 raft_consensus.cc:740] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:31.468031 1881 raft_consensus.cc:385] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:31.468580 1881 raft_consensus.cc:740] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:31.468266 1883 consensus_queue.cc:260] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.468356 1882 consensus_queue.cc:260] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.470148 1883 ts_tablet_manager.cc:1434] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260501 13:57:31.471009 1883 tablet_bootstrap.cc:492] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:31.488071 1883 tablet_bootstrap.cc:654] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:31.492849 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 104708eb7fc0412dad0bd7e14eee9d13
I20260501 13:57:31.506676 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet 104708eb7fc0412dad0bd7e14eee9d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:31.515635 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 104708eb7fc0412dad0bd7e14eee9d13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104708eb7fc0412dad0bd7e14eee9d13 already in progress: creating tablet
I20260501 13:57:31.521265 1976 consensus_queue.cc:1048] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:31.535629 1882 ts_tablet_manager.cc:1434] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.071s user 0.006s sys 0.001s
I20260501 13:57:31.550391 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 104708eb7fc0412dad0bd7e14eee9d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:31.594305 1882 tablet_bootstrap.cc:492] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:31.601384 1882 tablet_bootstrap.cc:654] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:31.604074 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 104708eb7fc0412dad0bd7e14eee9d13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104708eb7fc0412dad0bd7e14eee9d13 already in progress: creating tablet
I20260501 13:57:31.604929 1976 raft_consensus.cc:493] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:31.605417 1976 raft_consensus.cc:515] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.608145 1976 leader_election.cc:290] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
W20260501 13:57:31.561399 1840 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.1.16.190:45953, user_credentials={real_user=slave}} blocked reactor thread for 363771us
I20260501 13:57:31.616995 1124 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4b9a916d6a754ac7b38819b39d0b1671" } modify_external_catalogs: true
I20260501 13:57:31.617647 1124 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4b9a916d6a754ac7b38819b39d0b1671" } modify_external_catalogs: true
I20260501 13:57:31.624066 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_183071821cf844028ee50b8498ff77a3" } new_table_name: "default.Table_041fce4b819d4a4c80c58137b4213234" modify_external_catalogs: true
I20260501 13:57:31.626706 1123 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_37f461bb7f1c453fbeeab99b079891fb" } new_table_name: "default.Table_d6a9c181aaa946dc9fd2c647d50ce3ea" modify_external_catalogs: true
W20260501 13:57:31.629868 1119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260501 13:57:31.632153 1883 tablet_bootstrap.cc:492] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:31.632565 1883 ts_tablet_manager.cc:1403] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.162s user 0.042s sys 0.024s
I20260501 13:57:31.636160 1883 raft_consensus.cc:359] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.636782 1883 raft_consensus.cc:385] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:31.637037 1883 raft_consensus.cc:740] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:31.637701 1883 consensus_queue.cc:260] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.640743 1883 ts_tablet_manager.cc:1434] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260501 13:57:31.646021 1883 tablet_bootstrap.cc:492] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:31.610488 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ff05404061447199d936d005afda653" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:31.652958 1762 tablet_service.cc:1511] Processing CreateTablet for tablet a9672aab85c64d548a4f6583a1e9af20 (DEFAULT_TABLE table=default.Table_4f8f3a483f7845b0b04a11a999253c87 [id=787286aa1fb44c45827992c1cf240e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.654022 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9672aab85c64d548a4f6583a1e9af20. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:31.654098 1719 leader_election.cc:343] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:31.655722 1131 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_79262cb3701444fe8ad303b4bbe08238 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 1 got 0
I20260501 13:57:31.656159 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 (DEFAULT_TABLE table=default.Table_183071821cf844028ee50b8498ff77a3 [id=797990ef44df4a5685fa58bd4c3547bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.657073 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.659550 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet adc47b3b2b084d33a12bd28cfad27382 (table default.Table_c4edbd4ba4c54b1e9f9909c9badd7c54 [id=1de866dfd0d145e7a13e5ae1f367524b]). Expected version 1 got 0
I20260501 13:57:31.660108 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 2ff05404061447199d936d005afda653 (table default.Table_26541c8dd6a347a78e26e6cb55fa3b06 [id=15dd9506496343159cc3f4de3afcef5e]). Expected version 1 got 0
I20260501 13:57:31.660590 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 5e86d6c935d6444b8a953e7be1bdfa51 (table default.Table_299166301fa440eca9a509c07bde6ab3 [id=0f9c1230865848adaa4e5ea744c31c8b]). Expected version 2 got 0
I20260501 13:57:31.678567 1976 raft_consensus.cc:493] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:31.679477 1976 raft_consensus.cc:515] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:31.686955 1123 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ba63c153ef04db891cae46a71439455 consensus state has no leader
I20260501 13:57:31.687250 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c2d5336bfec43dbb38ce7071834bca3" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:31.688267 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 consensus state has no leader
W20260501 13:57:31.688439 1719 leader_election.cc:343] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:31.684109 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet e41b43329e054442bc8697cc28161954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:31.689275 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 89e2c22052d04fe1a21bcb3323385111
I20260501 13:57:31.689405 1899 tablet_replica.cc:333] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:31.690200 1899 raft_consensus.cc:2243] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:31.690660 1899 raft_consensus.cc:2272] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:31.691303 1882 tablet_bootstrap.cc:492] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:31.691668 1882 ts_tablet_manager.cc:1403] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.155s user 0.003s sys 0.007s
I20260501 13:57:31.692418 1899 ts_tablet_manager.cc:1916] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:31.694074 1882 raft_consensus.cc:359] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.694650 1883 tablet_bootstrap.cc:654] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:31.694696 1882 raft_consensus.cc:385] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:31.694829 1760 tablet_service.cc:1558] Processing DeleteTablet for tablet 89e2c22052d04fe1a21bcb3323385111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:31.694954 1882 raft_consensus.cc:740] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:31.702571 1976 leader_election.cc:290] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
W20260501 13:57:31.703182 1119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.188:38815, user_credentials={real_user=slave}} blocked reactor thread for 73324.7us
I20260501 13:57:31.705302 1606 tablet_service.cc:1511] Processing CreateTablet for tablet a9672aab85c64d548a4f6583a1e9af20 (DEFAULT_TABLE table=default.Table_4f8f3a483f7845b0b04a11a999253c87 [id=787286aa1fb44c45827992c1cf240e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.706430 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9672aab85c64d548a4f6583a1e9af20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.708478 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 89e2c22052d04fe1a21bcb3323385111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:31.708705 1119 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 5e86d6c935d6444b8a953e7be1bdfa51 (table default.Table_299166301fa440eca9a509c07bde6ab3 [id=0f9c1230865848adaa4e5ea744c31c8b], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5e86d6c935d6444b8a953e7be1bdfa51 consensus state has no leader
W20260501 13:57:31.709287 1119 consensus_peers.cc:597] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b -> Peer 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815): Couldn't send request to peer 50c0932256e74399a537670bb424a1b7. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260501 13:57:31.710394 1603 tablet_service.cc:1511] Processing CreateTablet for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 (DEFAULT_TABLE table=default.Table_183071821cf844028ee50b8498ff77a3 [id=797990ef44df4a5685fa58bd4c3547bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.711444 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:31.719995 1719 consensus_peers.cc:597] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:31.469278 1881 consensus_queue.cc:260] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.721395 1883 tablet_bootstrap.cc:492] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:31.721822 1883 ts_tablet_manager.cc:1403] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.005s
I20260501 13:57:31.722311 1881 ts_tablet_manager.cc:1434] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.467s user 0.006s sys 0.008s
I20260501 13:57:31.724056 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 96e937b3a0b5451e8eaf4d5c47c17620 (DEFAULT_TABLE table=default.Table_bd1dfe371f64485fb2a9b88dca4dd231 [id=05998ead56ed454dab9a71ef2ac9cb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.725262 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e937b3a0b5451e8eaf4d5c47c17620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.727248 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 406e1af8b7d9488f88697206daa87c05 (DEFAULT_TABLE table=default.Table_609f259197a542caaa0dd2fdc258b986 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.728358 1457 tablet_service.cc:1558] Processing DeleteTablet for tablet 104708eb7fc0412dad0bd7e14eee9d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:31.730436 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 59508ee6bc6b47959d09f8f5aab89804 (DEFAULT_TABLE table=default.Table_af01b0d8a23647a48ebf1ea88cb91b16 [id=87d2fd558d0e47da850bba54488295d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.725277 1883 raft_consensus.cc:359] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.734942 1883 raft_consensus.cc:385] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:31.735250 1883 raft_consensus.cc:740] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:31.735863 1883 consensus_queue.cc:260] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.737808 1883 ts_tablet_manager.cc:1434] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260501 13:57:31.738663 1883 tablet_bootstrap.cc:492] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:31.739562 1458 consensus_meta.cc:287] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.698s user 0.008s sys 0.001s
W20260501 13:57:31.742600 1418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 269937us
I20260501 13:57:31.745376 1984 raft_consensus.cc:493] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:31.745752 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc6b0cfd20174d7680f8981bb850c50f" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:31.746285 1483 raft_consensus.cc:3060] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:31.745900 1984 raft_consensus.cc:515] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.747782 1883 tablet_bootstrap.cc:654] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:31.747965 1984 leader_election.cc:290] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:31.749693 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e15d66bf6147a08604daf2a9c53322" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:31.750316 1783 raft_consensus.cc:2468] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:31.751560 1563 leader_election.cc:304] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:31.752553 1984 raft_consensus.cc:2804] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:31.752954 1984 raft_consensus.cc:493] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:31.753335 1984 raft_consensus.cc:3060] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:31.754283 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 406e1af8b7d9488f88697206daa87c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.755766 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59508ee6bc6b47959d09f8f5aab89804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.760536 1984 raft_consensus.cc:515] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.763703 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e15d66bf6147a08604daf2a9c53322" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:31.764286 1783 raft_consensus.cc:3060] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:31.764153 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_52acb3d1a0d1450bb41440f4de75ac91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:31.765849 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52acb3d1a0d1450bb41440f4de75ac91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:31.778414 1783 raft_consensus.cc:2468] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:31.779803 1563 leader_election.cc:304] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:31.780581 2039 raft_consensus.cc:2804] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:31.781030 2039 raft_consensus.cc:697] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:31.781751 2039 consensus_queue.cc:237] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:31.783296 1465 consensus_meta.cc:287] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.523s user 0.005s sys 0.000s
W20260501 13:57:31.798911 1719 consensus_peers.cc:597] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:31.808224 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 96e937b3a0b5451e8eaf4d5c47c17620 (DEFAULT_TABLE table=default.Table_bd1dfe371f64485fb2a9b88dca4dd231 [id=05998ead56ed454dab9a71ef2ac9cb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.808357 1457 tablet_service.cc:1511] Processing CreateTablet for tablet a9672aab85c64d548a4f6583a1e9af20 (DEFAULT_TABLE table=default.Table_4f8f3a483f7845b0b04a11a999253c87 [id=787286aa1fb44c45827992c1cf240e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.809525 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 59508ee6bc6b47959d09f8f5aab89804 (DEFAULT_TABLE table=default.Table_af01b0d8a23647a48ebf1ea88cb91b16 [id=87d2fd558d0e47da850bba54488295d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.809494 1457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9672aab85c64d548a4f6583a1e9af20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.811281 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59508ee6bc6b47959d09f8f5aab89804. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:31.811445 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 104708eb7fc0412dad0bd7e14eee9d13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104708eb7fc0412dad0bd7e14eee9d13 already in progress: creating tablet
I20260501 13:57:31.813690 2035 consensus_queue.cc:1048] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:57:31.815533 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adc47b3b2b084d33a12bd28cfad27382" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:31.827145 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 (DEFAULT_TABLE table=default.Table_183071821cf844028ee50b8498ff77a3 [id=797990ef44df4a5685fa58bd4c3547bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.828230 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.828293 1460 tablet_service.cc:1511] Processing CreateTablet for tablet 406e1af8b7d9488f88697206daa87c05 (DEFAULT_TABLE table=default.Table_609f259197a542caaa0dd2fdc258b986 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:31.832782 1119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.188:38815, user_credentials={real_user=slave}} blocked reactor thread for 123520us
I20260501 13:57:31.834425 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_641d4d711c4d49319daa2cb0cf2714c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:31.834853 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_6ee51a9fc8ae4340b14c506cf9c450c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:31.836656 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 89e2c22052d04fe1a21bcb3323385111 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89e2c22052d04fe1a21bcb3323385111 already in progress: creating tablet
W20260501 13:57:31.836733 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_641d4d711c4d49319daa2cb0cf2714c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:31.836959 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ee51a9fc8ae4340b14c506cf9c450c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:31.839435 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_2a23504aef0e48beaf3541c53fda054e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:31.840860 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a23504aef0e48beaf3541c53fda054e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:31.843808 1984 leader_election.cc:290] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:31.843721 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 406e1af8b7d9488f88697206daa87c05 (DEFAULT_TABLE table=default.Table_609f259197a542caaa0dd2fdc258b986 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.844643 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 406e1af8b7d9488f88697206daa87c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.855356 1901 raft_consensus.cc:493] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:31.855824 1901 raft_consensus.cc:515] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.856479 1125 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6452fc6a9103447c9239a13e344f0c5a" } modify_external_catalogs: true
I20260501 13:57:31.857004 1125 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6452fc6a9103447c9239a13e344f0c5a" } modify_external_catalogs: true
I20260501 13:57:31.695511 1882 consensus_queue.cc:260] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.859303 1882 ts_tablet_manager.cc:1434] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.167s user 0.006s sys 0.000s
I20260501 13:57:31.860718 1882 tablet_bootstrap.cc:492] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:31.867463 1976 raft_consensus.cc:493] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 13:57:31.868875 1717 leader_election.cc:343] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:31.869298 1717 leader_election.cc:304] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:31.882998 1901 leader_election.cc:290] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:31.883513 1901 raft_consensus.cc:2749] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:31.863868 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 96e937b3a0b5451e8eaf4d5c47c17620 (DEFAULT_TABLE table=default.Table_bd1dfe371f64485fb2a9b88dca4dd231 [id=05998ead56ed454dab9a71ef2ac9cb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.886384 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 59508ee6bc6b47959d09f8f5aab89804 (DEFAULT_TABLE table=default.Table_af01b0d8a23647a48ebf1ea88cb91b16 [id=87d2fd558d0e47da850bba54488295d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.887410 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59508ee6bc6b47959d09f8f5aab89804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.887974 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d2f196f44be4fbc8059901380b71271" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:31.889223 1719 leader_election.cc:343] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:31.891130 1630 raft_consensus.cc:3060] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:31.898847 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_445251ee80624ea789b17eee4cec7834" } new_table_name: "default.Table_f021eee779424a1d914441c79cc1e765" modify_external_catalogs: true
I20260501 13:57:31.899565 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e937b3a0b5451e8eaf4d5c47c17620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.902921 1128 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4275f119eec547c28a46b8d5f4faa0f8" } modify_external_catalogs: true
I20260501 13:57:31.903419 1128 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4275f119eec547c28a46b8d5f4faa0f8" } modify_external_catalogs: true
I20260501 13:57:31.908119 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_d6a9c181aaa946dc9fd2c647d50ce3ea" } new_table_name: "default.Table_66a4e496efa34deaa6c793a210a510ff" modify_external_catalogs: true
I20260501 13:57:31.843546 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e937b3a0b5451e8eaf4d5c47c17620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.852840 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet 89e2c22052d04fe1a21bcb3323385111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:31.865317 1485 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.051s sys 0.053s
I20260501 13:57:31.868331 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ff05404061447199d936d005afda653" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:31.869066 1481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c2d5336bfec43dbb38ce7071834bca3" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:31.912750 1976 raft_consensus.cc:515] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.914690 1976 leader_election.cc:290] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:31.889904 1881 tablet_bootstrap.cc:492] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:31.934486 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a563bccc7b046658ed133109f676528" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:31.936873 1719 leader_election.cc:343] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:31.941181 1882 tablet_bootstrap.cc:654] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:31.946069 1899 ts_tablet_manager.cc:1929] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260501 13:57:31.950423 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ba63c153ef04db891cae46a71439455 consensus state has no leader
I20260501 13:57:31.953361 1127 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_79262cb3701444fe8ad303b4bbe08238 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 1 got 0
I20260501 13:57:31.957862 1990 consensus_queue.cc:1048] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:31.962534 1421 consensus_peers.cc:597] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:31.964200 1899 log.cc:1199] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/e41b43329e054442bc8697cc28161954
I20260501 13:57:31.964490 1127 catalog_manager.cc:5671] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:31.965392 1899 ts_tablet_manager.cc:1950] T e41b43329e054442bc8697cc28161954 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:31.966961 1127 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_79262cb3701444fe8ad303b4bbe08238 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 1 got 0
I20260501 13:57:31.967711 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a563bccc7b046658ed133109f676528 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:31.969789 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 3eb6d30bed2a410db8b62fdff54a5fac (DEFAULT_TABLE table=default.Table_52acb3d1a0d1450bb41440f4de75ac91 [id=7b430735e8b640dc84cf92f308f20362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.977265 1128 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 45907da8b56f4fe887253c77ae12f9a3
I20260501 13:57:31.979810 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 3eb6d30bed2a410db8b62fdff54a5fac (DEFAULT_TABLE table=default.Table_52acb3d1a0d1450bb41440f4de75ac91 [id=7b430735e8b640dc84cf92f308f20362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:31.980208 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet 45907da8b56f4fe887253c77ae12f9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:31.980911 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eb6d30bed2a410db8b62fdff54a5fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.988529 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eb6d30bed2a410db8b62fdff54a5fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:31.992457 1883 tablet_bootstrap.cc:492] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:31.992884 1883 ts_tablet_manager.cc:1403] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.254s user 0.018s sys 0.007s
I20260501 13:57:31.995342 1883 raft_consensus.cc:359] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:31.995971 1883 raft_consensus.cc:385] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:31.996220 1883 raft_consensus.cc:740] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:32.003153 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_50ac7f827cc144898d7bf1f2784aea18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.005396 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50ac7f827cc144898d7bf1f2784aea18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.015519 1882 tablet_bootstrap.cc:492] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:32.017125 1882 ts_tablet_manager.cc:1403] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.157s user 0.008s sys 0.003s
I20260501 13:57:32.017704 1883 consensus_queue.cc:260] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.028532 1899 tablet_replica.cc:333] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:32.029441 1901 raft_consensus.cc:493] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:32.030190 1901 raft_consensus.cc:515] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.030658 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet e41b43329e054442bc8697cc28161954 (table default.Table_2edf812ced814de8acadfe87bcad5893 [id=694ff2769ea04445811ca8d8a92854fe]) successfully deleted
I20260501 13:57:32.032673 1901 leader_election.cc:290] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:32.035732 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 45907da8b56f4fe887253c77ae12f9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:32.036098 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7645593dc184dab9f677f9371a88354" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:32.036487 2043 tablet_replica.cc:333] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:32.037240 2043 raft_consensus.cc:2243] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:32.037686 2043 raft_consensus.cc:2272] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:32.039709 2043 ts_tablet_manager.cc:1916] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:32.043004 1882 raft_consensus.cc:359] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:31.939077 1485 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.922s user 0.052s sys 0.054s
I20260501 13:57:32.049497 1882 raft_consensus.cc:385] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:32.050832 1882 raft_consensus.cc:740] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:32.052810 1882 consensus_queue.cc:260] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.056094 1485 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.16.131:36337 (request call id 24) took 1045 ms. Trace:
I20260501 13:57:32.056493 1485 rpcz_store.cc:276] 0501 13:57:31.010411 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:31.010588 (+ 177us) service_pool.cc:224] Handling call
0501 13:57:31.015815 (+ 5227us) raft_consensus.cc:1471] Updating replica for 1 ops
0501 13:57:31.016664 (+ 849us) raft_consensus.cc:1534] Early marking committed up to index 0
0501 13:57:31.016693 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0501 13:57:31.017154 (+ 461us) log.cc:844] Serialized 26 byte log entry
0501 13:57:32.055699 (+1038545us) raft_consensus.cc:1645] Marking committed up to 0
0501 13:57:32.055754 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0501 13:57:32.055836 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0501 13:57:32.055875 (+ 39us) raft_consensus.cc:1685] finished
0501 13:57:32.055889 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0501 13:57:32.056012 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1031,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":161,"spinlock_wait_cycles":18816,"thread_start_us":1031483,"threads_started":1,"wal-append.queue_time_us":399173}
I20260501 13:57:32.060112 1125 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet eeae4369be954d9d9bad846484afb6ff
I20260501 13:57:32.062709 1481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e15d66bf6147a08604daf2a9c53322" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:32.098033 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_5d2b24ed68b54d58990aea3df63012ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.099551 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d2b24ed68b54d58990aea3df63012ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:32.099709 1719 leader_election.cc:343] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:32.100121 1719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 53509.9us
W20260501 13:57:32.099982 1717 leader_election.cc:343] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:32.104727 1717 leader_election.cc:304] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:32.105863 2053 raft_consensus.cc:2749] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:32.106763 1883 ts_tablet_manager.cc:1434] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.114s user 0.007s sys 0.000s
I20260501 13:57:32.107715 1883 tablet_bootstrap.cc:492] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:32.111320 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet eeae4369be954d9d9bad846484afb6ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:32.112401 1899 raft_consensus.cc:2243] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:32.069402 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d2f196f44be4fbc8059901380b71271" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:32.121505 1479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a563bccc7b046658ed133109f676528" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:32.103279 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 406e1af8b7d9488f88697206daa87c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.126691 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_6f9bd93befbf45c0a7415d32ce58d7fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.128903 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f9bd93befbf45c0a7415d32ce58d7fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.128038 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e15d66bf6147a08604daf2a9c53322" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:32.136626 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_ddd647c4319b431f9d4e874bb528755d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.138409 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddd647c4319b431f9d4e874bb528755d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:32.138994 1717 leader_election.cc:343] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:32.139439 1717 leader_election.cc:304] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:32.141902 1901 raft_consensus.cc:2749] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:32.143656 1483 raft_consensus.cc:2468] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 1.
I20260501 13:57:32.147270 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet b8d0f6d1f38b428291dba22015ac6959 (table default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba]). Expected version 1 got 0
I20260501 13:57:32.150879 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_50ac7f827cc144898d7bf1f2784aea18" } new_table_name: "default.Table_5d025c3dd5ab48069a6bd83f8fa4c73f" modify_external_catalogs: true
I20260501 13:57:32.151921 1883 tablet_bootstrap.cc:654] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:32.164386 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_f2b43b1500494a4bac22cfa9b776b136" } new_table_name: "default.Table_427f541417f845efb488354adcc40139" modify_external_catalogs: true
W20260501 13:57:32.168043 1421 consensus_peers.cc:597] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:32.178483 1444 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57810 (request call id 2) took 1315 ms. Trace:
I20260501 13:57:32.179087 1444 rpcz_store.cc:276] 0501 13:57:30.863019 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:30.864387 (+ 1368us) service_pool.cc:224] Handling call
0501 13:57:31.062513 (+198126us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b5c0006040c. stack: 00007f9e73a05c83 00007f9e73a05b80 00007f9e73576574 000000000050e094 000000000050e029 00007f9e7ae8d671 00007f9e7b02e889 00007f9e7af19810 00007f9e7bb27d70 00007f9e7a2ac104 00007f9e7a2ac092 00007f9e7a2abfc1 00007f9e7a2abf3c 00007f9e7a2ab1f2 00007f9e787debcc 00007f9e787de006
0501 13:57:32.136044 (+1073531us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400020154. stack: 00007f9e73a05c83 00007f9e73a05b80 00007f9e73576574 000000000050e094 000000000050e029 00007f9e7aa54951 00007f9e7aae52d8 00007f9e7bb27d9d 00007f9e7a2ac104 00007f9e7a2ac092 00007f9e7a2abfc1 00007f9e7a2abf3c 00007f9e7a2ab1f2 00007f9e787debcc 00007f9e787de006 00007f9e787dd9af
0501 13:57:32.177937 (+ 41893us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2342764032}
I20260501 13:57:32.183521 1445 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57810 (request call id 3) took 1319 ms. Trace:
I20260501 13:57:32.184029 1445 rpcz_store.cc:276] 0501 13:57:30.863416 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:30.868304 (+ 4888us) service_pool.cc:224] Handling call
0501 13:57:31.024132 (+155828us) spinlock_profiling.cc:243] Waited 9.19 ms on lock 0x7b5c0006078c. stack: 00007f9e73a05c83 00007f9e73a05b80 00007f9e73576574 000000000050e094 000000000050e029 00007f9e7ae8d671 00007f9e7b02e889 00007f9e7af19810 00007f9e7bb27d70 00007f9e7a2ac104 00007f9e7a2ac092 00007f9e7a2abfc1 00007f9e7a2abf3c 00007f9e7a2ab1f2 00007f9e787debcc 00007f9e787de006
0501 13:57:32.133628 (+1109496us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x7b5400020154. stack: 00007f9e73a05c83 00007f9e73a05b80 00007f9e73576574 000000000050e094 000000000050e029 00007f9e7aa54951 00007f9e7aae52d8 00007f9e7bb27d9d 00007f9e7a2ac104 00007f9e7a2ac092 00007f9e7a2abfc1 00007f9e7a2abf3c 00007f9e7a2ab1f2 00007f9e787debcc 00007f9e787de006 00007f9e787dd9af
0501 13:57:32.183058 (+ 49430us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2306987648}
I20260501 13:57:32.186370 1458 tablet_service.cc:1558] Processing DeleteTablet for tablet 45907da8b56f4fe887253c77ae12f9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:32.186623 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 3eb6d30bed2a410db8b62fdff54a5fac (DEFAULT_TABLE table=default.Table_52acb3d1a0d1450bb41440f4de75ac91 [id=7b430735e8b640dc84cf92f308f20362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.189404 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet eeae4369be954d9d9bad846484afb6ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:32.192620 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7645593dc184dab9f677f9371a88354" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:32.207530 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eb6d30bed2a410db8b62fdff54a5fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.212623 1899 pending_rounds.cc:70] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: Trying to abort 1 pending ops.
I20260501 13:57:32.212988 1899 pending_rounds.cc:77] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229213385428992 op_type: NO_OP noop_request { }
I20260501 13:57:32.213377 1899 raft_consensus.cc:2889] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:32.213713 1899 raft_consensus.cc:2272] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:32.227872 1881 tablet_bootstrap.cc:654] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:32.097409 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet eeae4369be954d9d9bad846484afb6ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:32.099802 2043 ts_tablet_manager.cc:1929] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:32.231647 2043 log.cc:1199] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/45907da8b56f4fe887253c77ae12f9a3
I20260501 13:57:32.231542 2039 raft_consensus.cc:493] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:32.232389 2043 ts_tablet_manager.cc:1950] T 45907da8b56f4fe887253c77ae12f9a3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:32.232144 2039 raft_consensus.cc:515] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.235224 2039 leader_election.cc:290] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:32.188913 1564 leader_election.cc:343] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:32.236088 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 47230.9us
I20260501 13:57:32.236330 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5d4064154c744198fa438cd3e3b7288" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
W20260501 13:57:32.236829 1564 leader_election.cc:343] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:32.244232 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 45907da8b56f4fe887253c77ae12f9a3 (table default.Table_4275f119eec547c28a46b8d5f4faa0f8 [id=5cd8cae1c55f41aa809960d1cecddec3]) successfully deleted
W20260501 13:57:32.245057 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet eeae4369be954d9d9bad846484afb6ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeae4369be954d9d9bad846484afb6ff already in progress: creating tablet
I20260501 13:57:32.249745 1882 ts_tablet_manager.cc:1434] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.227s user 0.004s sys 0.001s
I20260501 13:57:32.262498 1123 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_83779be1d30c4839999b9217ce2e4a20" } modify_external_catalogs: true
I20260501 13:57:32.263181 1123 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_83779be1d30c4839999b9217ce2e4a20" } modify_external_catalogs: true
I20260501 13:57:32.264814 1131 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_8999f52c517a462a9845800aba73657e" } modify_external_catalogs: true
I20260501 13:57:32.265305 1131 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_8999f52c517a462a9845800aba73657e" } modify_external_catalogs: true
I20260501 13:57:32.270234 1132 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_f2b43b1500494a4bac22cfa9b776b136 [id=0dc2b91768f84b4d93b18f304d31883e]). Expected version 1 got 0
I20260501 13:57:32.271138 1899 ts_tablet_manager.cc:1916] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:32.275025 1882 tablet_bootstrap.cc:492] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:32.277679 1761 tablet_service.cc:1511] Processing CreateTablet for tablet ec4bd4cb1e70472aa09fc93afc3b1501 (DEFAULT_TABLE table=default.Table_6ee51a9fc8ae4340b14c506cf9c450c2 [id=e61fe8a70f66497db17db0afacc35100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.278815 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4bd4cb1e70472aa09fc93afc3b1501. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:32.239907 1461 consensus_meta.cc:287] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.504s user 0.006s sys 0.001s
I20260501 13:57:32.245078 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5d4064154c744198fa438cd3e3b7288" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:32.201290 1418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 138661us
I20260501 13:57:32.284895 1606 tablet_service.cc:1511] Processing CreateTablet for tablet ec4bd4cb1e70472aa09fc93afc3b1501 (DEFAULT_TABLE table=default.Table_6ee51a9fc8ae4340b14c506cf9c450c2 [id=e61fe8a70f66497db17db0afacc35100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.285144 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 0669a229d6d54eacac9889ad18e06f30 (DEFAULT_TABLE table=default.Table_641d4d711c4d49319daa2cb0cf2714c6 [id=535a372705e4460ab657255e3f1be962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.286322 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4bd4cb1e70472aa09fc93afc3b1501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.290933 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0669a229d6d54eacac9889ad18e06f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.291110 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 (DEFAULT_TABLE table=default.Table_2a23504aef0e48beaf3541c53fda054e [id=0e61d9e75f4c4b20a849a0540c83bd18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.292315 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9abb39c8b44d4b4c8b69751a4cdd6668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.296272 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 (DEFAULT_TABLE table=default.Table_2a23504aef0e48beaf3541c53fda054e [id=0e61d9e75f4c4b20a849a0540c83bd18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.299120 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 0669a229d6d54eacac9889ad18e06f30 (DEFAULT_TABLE table=default.Table_641d4d711c4d49319daa2cb0cf2714c6 [id=535a372705e4460ab657255e3f1be962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.300729 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9abb39c8b44d4b4c8b69751a4cdd6668. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:32.302237 1128 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0356b19f0aa4197a41cc66d4440bffb has no consensus state
I20260501 13:57:32.302744 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0669a229d6d54eacac9889ad18e06f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.305711 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_cf80c30d6b014d56aab7e294515801d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.307189 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf80c30d6b014d56aab7e294515801d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.325083 1131 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet b98187c7843e42b09d55c851117c4ac8
I20260501 13:57:32.326332 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_291632318d004a2d91a9f8d79137ef4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.327864 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_291632318d004a2d91a9f8d79137ef4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.330719 2064 raft_consensus.cc:493] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:32.330960 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet b98187c7843e42b09d55c851117c4ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:32.331173 2064 raft_consensus.cc:515] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.332952 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet b98187c7843e42b09d55c851117c4ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:32.333081 2064 leader_election.cc:290] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
W20260501 13:57:32.334333 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet b98187c7843e42b09d55c851117c4ac8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b98187c7843e42b09d55c851117c4ac8 already in progress: creating tablet
I20260501 13:57:32.335707 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c33a1f423bdb44f5843daffe8f73e74a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:32.336428 1123 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet b507a4d4c9c54bc68d92d9286794a264
W20260501 13:57:32.338330 1719 leader_election.cc:343] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:32.338734 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet b507a4d4c9c54bc68d92d9286794a264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:32.339996 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet b507a4d4c9c54bc68d92d9286794a264 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b507a4d4c9c54bc68d92d9286794a264 already in progress: creating tablet
I20260501 13:57:32.341638 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet b507a4d4c9c54bc68d92d9286794a264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:32.367698 1564 leader_election.cc:304] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:32.368492 2039 raft_consensus.cc:2749] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:32.372135 1717 leader_election.cc:343] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:32.372583 1717 leader_election.cc:304] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:32.373375 2064 raft_consensus.cc:2749] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:32.373700 2047 tablet_replica.cc:333] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:32.373914 2064 raft_consensus.cc:493] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:32.374516 2047 raft_consensus.cc:2243] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:32.374285 2064 raft_consensus.cc:515] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.376179 2064 leader_election.cc:290] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:32.376266 1464 tablet_service.cc:1511] Processing CreateTablet for tablet ec4bd4cb1e70472aa09fc93afc3b1501 (DEFAULT_TABLE table=default.Table_6ee51a9fc8ae4340b14c506cf9c450c2 [id=e61fe8a70f66497db17db0afacc35100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.377321 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4bd4cb1e70472aa09fc93afc3b1501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.378446 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e86d6c935d6444b8a953e7be1bdfa51" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:32.379696 1719 leader_election.cc:343] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:32.379516 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 0669a229d6d54eacac9889ad18e06f30 (DEFAULT_TABLE table=default.Table_641d4d711c4d49319daa2cb0cf2714c6 [id=535a372705e4460ab657255e3f1be962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.380656 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_945a3cdaaa824caab261338095683b43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.382129 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_945a3cdaaa824caab261338095683b43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.382921 1883 tablet_bootstrap.cc:492] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:32.383302 1883 ts_tablet_manager.cc:1403] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.276s user 0.026s sys 0.028s
I20260501 13:57:32.385535 1883 raft_consensus.cc:359] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.386132 1883 raft_consensus.cc:385] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:32.386358 1883 raft_consensus.cc:740] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:32.386932 1883 consensus_queue.cc:260] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.390741 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0669a229d6d54eacac9889ad18e06f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.400738 1883 ts_tablet_manager.cc:1434] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260501 13:57:32.406580 1717 leader_election.cc:304] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:32.407691 2064 raft_consensus.cc:2804] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:32.408164 2064 raft_consensus.cc:697] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Running, Role: LEADER
I20260501 13:57:32.408317 1132 master_service.cc:954] ReplaceTablet: received request to replace tablet 104708eb7fc0412dad0bd7e14eee9d13 from {username='slave'} at 127.0.0.1:41700
W20260501 13:57:32.405360 1457 consensus_meta.cc:287] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.539s user 0.003s sys 0.005s
I20260501 13:57:32.408869 2064 consensus_queue.cc:237] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:32.409471 1717 leader_election.cc:343] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:32.409890 1717 leader_election.cc:304] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:32.410789 1976 raft_consensus.cc:2749] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:32.412701 1458 tablet_service.cc:1511] Processing CreateTablet for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 (DEFAULT_TABLE table=default.Table_2a23504aef0e48beaf3541c53fda054e [id=0e61d9e75f4c4b20a849a0540c83bd18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.413817 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9abb39c8b44d4b4c8b69751a4cdd6668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.433140 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet b98187c7843e42b09d55c851117c4ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:32.433373 1132 master_service.cc:954] ReplaceTablet: received request to replace tablet 104708eb7fc0412dad0bd7e14eee9d13 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:32.434026 1883 tablet_bootstrap.cc:492] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:32.438028 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 104708eb7fc0412dad0bd7e14eee9d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:32.441905 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 67b0675398d04be08bd29638b4ba2083 (DEFAULT_TABLE table=default.Table_ddd647c4319b431f9d4e874bb528755d [id=d4ac3b610db54cb8aae381cbc882d18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.442867 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67b0675398d04be08bd29638b4ba2083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.444916 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 33a355897309420c97b1257afaefe8d1 (DEFAULT_TABLE table=default.Table_5d2b24ed68b54d58990aea3df63012ba [id=84b84a980ef34f0292ef7b9488a09b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.445824 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a355897309420c97b1257afaefe8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.447186 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 67b0675398d04be08bd29638b4ba2083 (DEFAULT_TABLE table=default.Table_ddd647c4319b431f9d4e874bb528755d [id=d4ac3b610db54cb8aae381cbc882d18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.447438 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 33a355897309420c97b1257afaefe8d1 (DEFAULT_TABLE table=default.Table_5d2b24ed68b54d58990aea3df63012ba [id=84b84a980ef34f0292ef7b9488a09b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.448350 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67b0675398d04be08bd29638b4ba2083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.448920 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a355897309420c97b1257afaefe8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.451460 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet b507a4d4c9c54bc68d92d9286794a264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:32.452059 1461 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57818 (request call id 70) took 1214 ms. Trace:
I20260501 13:57:32.451879 1123 catalog_manager.cc:5671] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: HEALTHY } } }
I20260501 13:57:32.454288 1609 tablet_service.cc:1511] Processing CreateTablet for tablet e0356b19f0aa4197a41cc66d4440bffb (DEFAULT_TABLE table=default.Table_5d025c3dd5ab48069a6bd83f8fa4c73f [id=d32e41f59f7d4b45842ea1b263eefbb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.455297 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0356b19f0aa4197a41cc66d4440bffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.452376 1461 rpcz_store.cc:276] 0501 13:57:31.237173 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:31.237889 (+ 716us) service_pool.cc:224] Handling call
0501 13:57:31.239830 (+ 1941us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0501 13:57:31.239912 (+ 82us) ts_tablet_manager.cc:682] Creating new metadata...
0501 13:57:31.736078 (+496166us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:32.395352 (+659274us) tablet_replica.cc:206] Creating consensus instance
0501 13:57:32.451997 (+ 56645us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52451}
I20260501 13:57:32.461196 1763 tablet_service.cc:1511] Processing CreateTablet for tablet e0356b19f0aa4197a41cc66d4440bffb (DEFAULT_TABLE table=default.Table_5d025c3dd5ab48069a6bd83f8fa4c73f [id=d32e41f59f7d4b45842ea1b263eefbb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.462146 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0356b19f0aa4197a41cc66d4440bffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.468494 1459 tablet_service.cc:1511] Processing CreateTablet for tablet e0356b19f0aa4197a41cc66d4440bffb (DEFAULT_TABLE table=default.Table_5d025c3dd5ab48069a6bd83f8fa4c73f [id=d32e41f59f7d4b45842ea1b263eefbb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.469439 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0356b19f0aa4197a41cc66d4440bffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.469197 1758 tablet_service.cc:1511] Processing CreateTablet for tablet 39cd02738e5f4ecfbfa06d5541614ffd (DEFAULT_TABLE table=default.Table_6f9bd93befbf45c0a7415d32ce58d7fb [id=edb6ad6e0b444d3b8930c43ce4082a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.472931 1883 tablet_bootstrap.cc:654] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:32.475472 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39cd02738e5f4ecfbfa06d5541614ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.476826 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 39cd02738e5f4ecfbfa06d5541614ffd (DEFAULT_TABLE table=default.Table_6f9bd93befbf45c0a7415d32ce58d7fb [id=edb6ad6e0b444d3b8930c43ce4082a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.477802 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39cd02738e5f4ecfbfa06d5541614ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.478217 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 39cd02738e5f4ecfbfa06d5541614ffd (DEFAULT_TABLE table=default.Table_6f9bd93befbf45c0a7415d32ce58d7fb [id=edb6ad6e0b444d3b8930c43ce4082a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.479148 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39cd02738e5f4ecfbfa06d5541614ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.486708 1463 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57818 (request call id 72) took 1189 ms. Trace:
I20260501 13:57:32.487080 1463 rpcz_store.cc:276] 0501 13:57:31.297404 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:31.398729 (+101325us) service_pool.cc:224] Handling call
0501 13:57:31.400174 (+ 1445us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0501 13:57:31.400257 (+ 83us) ts_tablet_manager.cc:682] Creating new metadata...
0501 13:57:31.823603 (+423346us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:32.191155 (+367552us) tablet_replica.cc:206] Creating consensus instance
0501 13:57:32.486634 (+295479us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 13:57:32.488783 1717 leader_election.cc:343] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:32.489188 1717 leader_election.cc:304] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:32.489976 1901 raft_consensus.cc:2749] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:32.494271 1783 raft_consensus.cc:1275] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Refusing update from remote peer 8f90b7ff170b4eadb27abc9ef866ffcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:57:32.495878 2039 consensus_queue.cc:1048] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:32.498944 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 67b0675398d04be08bd29638b4ba2083 (DEFAULT_TABLE table=default.Table_ddd647c4319b431f9d4e874bb528755d [id=d4ac3b610db54cb8aae381cbc882d18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.501965 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67b0675398d04be08bd29638b4ba2083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.504101 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_8a77fe75f8044d75a97b45dd4d94216f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.505568 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a77fe75f8044d75a97b45dd4d94216f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.507701 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6be94aa0ac634ce19d10a1c97e33fd5e" } new_table_name: "default.Table_f9c99e1d322d44258e1845eeb858e579" modify_external_catalogs: true
I20260501 13:57:32.510022 1882 tablet_bootstrap.cc:654] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:32.519802 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4f8f3a483f7845b0b04a11a999253c87" } new_table_name: "default.Table_bc1aee595536420799f8a28881253c09" modify_external_catalogs: true
I20260501 13:57:32.532099 1883 tablet_bootstrap.cc:492] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:32.539402 1125 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_2a23504aef0e48beaf3541c53fda054e" } modify_external_catalogs: true
I20260501 13:57:32.544766 1123 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c4edbd4ba4c54b1e9f9909c9badd7c54" } modify_external_catalogs: true
I20260501 13:57:32.545789 1125 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_2a23504aef0e48beaf3541c53fda054e" } modify_external_catalogs: true
I20260501 13:57:32.545925 1123 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c4edbd4ba4c54b1e9f9909c9badd7c54" } modify_external_catalogs: true
W20260501 13:57:32.550987 1128 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f9c92846d4e4255a1c8ce3835a0a18d consensus state has no leader
I20260501 13:57:32.552510 2047 raft_consensus.cc:2272] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:32.554811 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_e69ec1df728e4af78640327a1b36ad4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.555557 1564 consensus_peers.cc:597] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260501 13:57:32.556392 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e69ec1df728e4af78640327a1b36ad4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:32.556429 1418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 129216us
I20260501 13:57:32.553435 1127 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_79262cb3701444fe8ad303b4bbe08238 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 1 got 0
I20260501 13:57:32.559378 1990 consensus_queue.cc:1048] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:32.562004 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9672aab85c64d548a4f6583a1e9af20 consensus state has no leader
I20260501 13:57:32.563486 2047 ts_tablet_manager.cc:1916] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:32.576808 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_46b8725e01684c90bcbbde7b5a9f4e0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.578305 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46b8725e01684c90bcbbde7b5a9f4e0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.579242 2077 tablet.cc:1724] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:32.595170 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 33a355897309420c97b1257afaefe8d1 (DEFAULT_TABLE table=default.Table_5d2b24ed68b54d58990aea3df63012ba [id=84b84a980ef34f0292ef7b9488a09b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.607511 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_56892e9ecd4b4b0b9133bc936b37c1c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:32.571622 2076 tablet.cc:1724] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:32.610322 1883 ts_tablet_manager.cc:1403] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.176s user 0.007s sys 0.003s
W20260501 13:57:32.612684 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56892e9ecd4b4b0b9133bc936b37c1c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.613255 1127 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_79262cb3701444fe8ad303b4bbe08238 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 1 got 0
I20260501 13:57:32.612794 1883 raft_consensus.cc:359] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.614427 1883 raft_consensus.cc:385] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:32.614710 1883 raft_consensus.cc:740] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:32.615346 1883 consensus_queue.cc:260] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.619133 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a355897309420c97b1257afaefe8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.634819 1990 raft_consensus.cc:493] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:32.635293 1990 raft_consensus.cc:515] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.636987 1990 leader_election.cc:290] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:32.640491 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf96cb4696704478a2064d6ce785d49d" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:32.644598 1421 leader_election.cc:343] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:32.648253 1883 ts_tablet_manager.cc:1434] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20260501 13:57:32.655515 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e86d6c935d6444b8a953e7be1bdfa51" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:32.657094 1418 leader_election.cc:343] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: bf96cb4696704478a2064d6ce785d49d
I20260501 13:57:32.676328 1630 raft_consensus.cc:3060] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:32.678504 1883 tablet_bootstrap.cc:492] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:32.659921 1460 consensus_meta.cc:287] T 406e1af8b7d9488f88697206daa87c05 P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.506s user 0.005s sys 0.007s
I20260501 13:57:32.688612 1883 tablet_bootstrap.cc:654] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:32.691254 1717 leader_election.cc:343] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:32.691658 1717 leader_election.cc:304] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:32.692691 2053 raft_consensus.cc:2749] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:32.658131 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c33a1f423bdb44f5843daffe8f73e74a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:32.715899 1132 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_79262cb3701444fe8ad303b4bbe08238 [id=db653ae15d0b4dada66cc4a7b2ac94bd] alter complete (version 1)
I20260501 13:57:32.718001 1882 tablet_bootstrap.cc:492] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:32.718011 1883 tablet_bootstrap.cc:492] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:32.718392 1882 ts_tablet_manager.cc:1403] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.444s user 0.003s sys 0.008s
I20260501 13:57:32.723433 2077 tablet.cc:1724] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260501 13:57:32.726850 1116 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (table default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7f9c92846d4e4255a1c8ce3835a0a18d consensus state has no leader
I20260501 13:57:32.731714 1607 tablet_service.cc:1511] Processing CreateTablet for tablet d361a2d82ecd456fb43f5e7fe3c83094 (DEFAULT_TABLE table=default.Table_945a3cdaaa824caab261338095683b43 [id=e5c5a3c3a1544b599c53a9b56a6c7297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.735615 1606 tablet_service.cc:1511] Processing CreateTablet for tablet e1d7a80ad4c4455a9ddee7a955e32635 (DEFAULT_TABLE table=default.Table_cf80c30d6b014d56aab7e294515801d5 [id=16dcfdbf585e45b29a6cdc6442d0990b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.738822 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d361a2d82ecd456fb43f5e7fe3c83094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.741400 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d7a80ad4c4455a9ddee7a955e32635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.742635 1882 raft_consensus.cc:359] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.751114 1882 raft_consensus.cc:385] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:32.751475 1123 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet adc47b3b2b084d33a12bd28cfad27382
I20260501 13:57:32.752599 1882 raft_consensus.cc:740] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:32.757670 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_5d025c3dd5ab48069a6bd83f8fa4c73f" } new_table_name: "default.Table_5be69bddaf9d4f6a8e87137a8a46e223" modify_external_catalogs: true
I20260501 13:57:32.760938 1125 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 9abb39c8b44d4b4c8b69751a4cdd6668
I20260501 13:57:32.767944 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_79262cb3701444fe8ad303b4bbe08238" } new_table_name: "default.Table_a685edff90a04b37ba8dafcaea3d7f55" modify_external_catalogs: true
I20260501 13:57:32.657850 1418 leader_election.cc:304] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:32.773499 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_e7f17dee34594eb3a711ffbf706d4e59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.775216 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7f17dee34594eb3a711ffbf706d4e59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.753513 1882 consensus_queue.cc:260] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.787521 1882 ts_tablet_manager.cc:1434] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.069s user 0.004s sys 0.001s
I20260501 13:57:32.790889 1990 raft_consensus.cc:2749] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:32.795291 1127 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet b8d0f6d1f38b428291dba22015ac6959 (table default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba]). Expected version 1 got 0
I20260501 13:57:32.795964 1127 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_427f541417f845efb488354adcc40139 [id=0dc2b91768f84b4d93b18f304d31883e]). Expected version 2 got 0
W20260501 13:57:32.803149 1418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 146130us
I20260501 13:57:32.719053 2076 tablet.cc:1724] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:32.814798 1883 ts_tablet_manager.cc:1403] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.137s user 0.010s sys 0.000s
I20260501 13:57:32.817099 1883 raft_consensus.cc:359] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.817721 1883 raft_consensus.cc:385] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:32.817967 1883 raft_consensus.cc:740] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
W20260501 13:57:32.829154 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0356b19f0aa4197a41cc66d4440bffb consensus state has no leader
I20260501 13:57:32.838032 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_51bf305a64b9456c8cb5a9c56799f4cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.841133 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51bf305a64b9456c8cb5a9c56799f4cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.842271 1883 consensus_queue.cc:260] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.844513 1883 ts_tablet_manager.cc:1434] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20260501 13:57:32.869182 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ee580bb9d3843f9bbeae8eef2c347ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:32.872949 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_1dd226531a7b4933a12d0d6c798491af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:32.874760 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dd226531a7b4933a12d0d6c798491af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.724053 1899 ts_tablet_manager.cc:1929] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:32.731473 1761 tablet_service.cc:1511] Processing CreateTablet for tablet d361a2d82ecd456fb43f5e7fe3c83094 (DEFAULT_TABLE table=default.Table_945a3cdaaa824caab261338095683b43 [id=e5c5a3c3a1544b599c53a9b56a6c7297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.880074 1899 log.cc:1199] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/89e2c22052d04fe1a21bcb3323385111
I20260501 13:57:32.880632 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d361a2d82ecd456fb43f5e7fe3c83094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:32.884032 1899 ts_tablet_manager.cc:1950] T 89e2c22052d04fe1a21bcb3323385111 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:32.886969 1127 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_a685edff90a04b37ba8dafcaea3d7f55 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 2 got 1
I20260501 13:57:32.887756 1127 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet b8d0f6d1f38b428291dba22015ac6959 (table default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba]). Expected version 1 got 0
I20260501 13:57:32.913156 1465 tablet_service.cc:1511] Processing CreateTablet for tablet d361a2d82ecd456fb43f5e7fe3c83094 (DEFAULT_TABLE table=default.Table_945a3cdaaa824caab261338095683b43 [id=e5c5a3c3a1544b599c53a9b56a6c7297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.754326 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet adc47b3b2b084d33a12bd28cfad27382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:32.763926 1760 tablet_service.cc:1558] Processing DeleteTablet for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:32.914711 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_89c5b57152224983bcb4b5ef5acd7af4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:32.808948 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 (DEFAULT_TABLE table=default.Table_291632318d004a2d91a9f8d79137ef4d [id=4668751f7bc649019547c38d0f8254c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.916055 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:32.916201 1418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 112157us
W20260501 13:57:32.916628 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89c5b57152224983bcb4b5ef5acd7af4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:32.739897 1758 tablet_service.cc:1511] Processing CreateTablet for tablet e1d7a80ad4c4455a9ddee7a955e32635 (DEFAULT_TABLE table=default.Table_cf80c30d6b014d56aab7e294515801d5 [id=16dcfdbf585e45b29a6cdc6442d0990b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.919272 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d7a80ad4c4455a9ddee7a955e32635. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:32.906114 1717 leader_election.cc:343] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:32.922851 1717 leader_election.cc:304] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:32.924419 2053 raft_consensus.cc:2749] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:32.924990 1457 tablet_service.cc:1511] Processing CreateTablet for tablet e1d7a80ad4c4455a9ddee7a955e32635 (DEFAULT_TABLE table=default.Table_cf80c30d6b014d56aab7e294515801d5 [id=16dcfdbf585e45b29a6cdc6442d0990b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.927521 1132 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet b8d0f6d1f38b428291dba22015ac6959 (table default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba]). Expected version 1 got 0
I20260501 13:57:32.929569 1132 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_427f541417f845efb488354adcc40139 [id=0dc2b91768f84b4d93b18f304d31883e]). Expected version 2 got 0
I20260501 13:57:32.938863 1883 tablet_bootstrap.cc:492] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:32.949651 1883 tablet_bootstrap.cc:654] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:32.952704 1901 raft_consensus.cc:493] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:32.953147 1901 raft_consensus.cc:515] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:32.959007 1131 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_a74b32dea54d43a2a8a1b6f2210f38d8" } modify_external_catalogs: true
I20260501 13:57:32.959534 1131 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_a74b32dea54d43a2a8a1b6f2210f38d8" } modify_external_catalogs: true
I20260501 13:57:32.963631 1460 tablet_service.cc:1558] Processing DeleteTablet for tablet adc47b3b2b084d33a12bd28cfad27382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:32.964151 1117 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 49752.7us
I20260501 13:57:32.965032 1460 tablet_service.cc:1558] Processing DeleteTablet for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:32.965350 1128 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_26541c8dd6a347a78e26e6cb55fa3b06" } modify_external_catalogs: true
I20260501 13:57:32.965827 1128 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_26541c8dd6a347a78e26e6cb55fa3b06" } modify_external_catalogs: true
I20260501 13:57:32.969329 1460 tablet_service.cc:1511] Processing CreateTablet for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 (DEFAULT_TABLE table=default.Table_291632318d004a2d91a9f8d79137ef4d [id=4668751f7bc649019547c38d0f8254c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:32.963654 1881 tablet_bootstrap.cc:492] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:32.979506 1881 ts_tablet_manager.cc:1403] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 1.257s user 0.022s sys 0.014s
I20260501 13:57:32.959528 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 81) took 1265 ms. Trace:
I20260501 13:57:32.805495 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:32.806680 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet adc47b3b2b084d33a12bd28cfad27382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:32.809664 1603 tablet_service.cc:1511] Processing CreateTablet for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 (DEFAULT_TABLE table=default.Table_291632318d004a2d91a9f8d79137ef4d [id=4668751f7bc649019547c38d0f8254c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.014827 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.014732 1901 leader_election.cc:290] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:32.963125 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee580bb9d3843f9bbeae8eef2c347ad" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:33.019549 1719 leader_election.cc:343] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:33.033454 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_1dd226531a7b4933a12d0d6c798491af" } new_table_name: "default.Table_aa47824eda84455d9d52df61514a253e" modify_external_catalogs: true
I20260501 13:57:33.043429 2039 raft_consensus.cc:493] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:33.043870 2039 raft_consensus.cc:515] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.045747 2039 leader_election.cc:290] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:33.059342 1882 tablet_bootstrap.cc:492] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:33.069177 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_5e1da2dc1631431c9ced9ebb18d903d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:33.071048 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e1da2dc1631431c9ced9ebb18d903d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:33.072140 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4b01fcca8d2a46ccba7050ab708d8a8d" } new_table_name: "default.Table_360decd489b74f4b8ee93b008e57c45b" modify_external_catalogs: true
I20260501 13:57:32.981961 1881 raft_consensus.cc:359] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.019572 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.022817 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee580bb9d3843f9bbeae8eef2c347ad" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:33.103125 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d361a2d82ecd456fb43f5e7fe3c83094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.082384 2092 tablet.cc:1724] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:33.042610 2089 tablet.cc:1724] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:33.054605 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8d0f6d1f38b428291dba22015ac6959" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:33.012619 1899 rpcz_store.cc:276] 0501 13:57:31.694568 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:31.694703 (+ 135us) service_pool.cc:224] Handling call
0501 13:57:32.028384 (+333681us) ts_tablet_manager.cc:1151] Deleting tablet 89e2c22052d04fe1a21bcb3323385111
0501 13:57:32.028431 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:32.212164 (+183733us) spinlock_profiling.cc:243] Waited 97.6 ms on lock 0x7b58000701d0. stack: 00007fa5fdee3c83 00007fa5fdee3b80 00007fa5fda54574 000000000050e094 000000000050e029 00007fa604f32951 00007fa604f2548a 00007fa604fa7827 00007fa604fd1765 00007fa6053f47bc 00007fa60604ccca 00007fa60607d990 00007fa606063efd 00007fa606063ea9 00007fa606063e39 00007fa606063e01
0501 13:57:32.514087 (+301923us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:32.723932 (+209845us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:32.959468 (+235536us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82991,"spinlock_wait_cycles":224586496,"tablet-delete.queue_time_us":333185}
I20260501 13:57:33.149196 1899 tablet_replica.cc:333] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:33.105214 1457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d7a80ad4c4455a9ddee7a955e32635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.103708 1881 raft_consensus.cc:385] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:33.154874 1881 raft_consensus.cc:740] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:33.155627 1881 consensus_queue.cc:260] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.157789 1881 ts_tablet_manager.cc:1434] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.178s user 0.000s sys 0.005s
I20260501 13:57:33.164913 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 89e2c22052d04fe1a21bcb3323385111 (table default.Table_4b9a916d6a754ac7b38819b39d0b1671 [id=36e8cb1c80cc45e7b4c501608c374af9]) successfully deleted
I20260501 13:57:33.170670 1901 raft_consensus.cc:493] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:33.176829 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8d0f6d1f38b428291dba22015ac6959" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
W20260501 13:57:33.177155 1717 leader_election.cc:343] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:33.177536 1717 leader_election.cc:304] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:33.182725 2053 raft_consensus.cc:2749] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:33.183948 2076 tablet.cc:1724] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:33.171196 1901 raft_consensus.cc:515] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.191617 1901 leader_election.cc:290] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:33.204864 1899 raft_consensus.cc:2243] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:33.205359 1899 raft_consensus.cc:2272] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:33.207160 1899 ts_tablet_manager.cc:1916] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:33.230975 1123 master_service.cc:954] ReplaceTablet: received request to replace tablet 1ba63c153ef04db891cae46a71439455 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:33.235503 1124 master_service.cc:954] ReplaceTablet: received request to replace tablet 921e5c3eaaa44c62abbc90f560049614 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:33.253226 1127 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_427f541417f845efb488354adcc40139 [id=0dc2b91768f84b4d93b18f304d31883e] alter complete (version 2)
I20260501 13:57:33.255276 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_b6780bd7a18f45d8b8ca54038933768f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:33.256739 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6780bd7a18f45d8b8ca54038933768f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:33.265822 1899 ts_tablet_manager.cc:1929] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:33.266191 1899 log.cc:1199] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/4a563bccc7b046658ed133109f676528
I20260501 13:57:33.266817 1899 ts_tablet_manager.cc:1950] T 4a563bccc7b046658ed133109f676528 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:33.268597 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 85) took 1301 ms. Trace:
I20260501 13:57:33.268967 1899 rpcz_store.cc:276] 0501 13:57:31.966883 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:31.967576 (+ 693us) service_pool.cc:224] Handling call
0501 13:57:33.149028 (+1181452us) ts_tablet_manager.cc:1151] Deleting tablet 4a563bccc7b046658ed133109f676528
0501 13:57:33.149077 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:33.230601 (+ 81524us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:33.265702 (+ 35101us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:33.268537 (+ 2835us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37565,"tablet-delete.queue_time_us":1174894}
I20260501 13:57:33.270366 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 4a563bccc7b046658ed133109f676528 (table default.Table_9e3d29db155f4fd2ba6229fc2318edb0 [id=08c0033dd652465797b8be473d0e427f]) successfully deleted
I20260501 13:57:33.271646 1899 tablet_replica.cc:333] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:33.192358 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "104708eb7fc0412dad0bd7e14eee9d13" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:33.288769 1881 tablet_bootstrap.cc:492] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:33.289529 1129 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_a685edff90a04b37ba8dafcaea3d7f55 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 2 got 0
I20260501 13:57:33.297551 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet bf96cb4696704478a2064d6ce785d49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:33.300542 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 (DEFAULT_TABLE table=default.Table_46b8725e01684c90bcbbde7b5a9f4e0e [id=39aeda29815d474196136813b5d811e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.301539 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.303114 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 38c3eb388d474d718554775d65f58d78 (DEFAULT_TABLE table=default.Table_8a77fe75f8044d75a97b45dd4d94216f [id=d6ca68cf0d954555b422b90e6502d6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.304180 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38c3eb388d474d718554775d65f58d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.316526 1456 tablet_service.cc:1511] Processing CreateTablet for tablet e2ac522f2e4c4ab48a9b7d211c82ed9e (DEFAULT_TABLE table=default.Table_e69ec1df728e4af78640327a1b36ad4b [id=34b2db06c0a84cecbc10ec705caa4656]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:33.322196 1125 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d399de014d74cab8df14f44787ca2f7 has no consensus state
W20260501 13:57:33.328859 1126 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d2f196f44be4fbc8059901380b71271 consensus state has no leader
I20260501 13:57:33.328984 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ac522f2e4c4ab48a9b7d211c82ed9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.279100 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 38c3eb388d474d718554775d65f58d78 (DEFAULT_TABLE table=default.Table_8a77fe75f8044d75a97b45dd4d94216f [id=d6ca68cf0d954555b422b90e6502d6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.281828 2091 tablet.cc:1719] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: Skipping requested alter to schema version 2, tablet already version 2
I20260501 13:57:33.285418 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 0c736959df334aee857dbaae0f7a1e1a (DEFAULT_TABLE table=default.Table_56892e9ecd4b4b0b9133bc936b37c1c4 [id=09422f53fb424bdbaa00282b1775947d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.286196 1758 tablet_service.cc:1511] Processing CreateTablet for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 (DEFAULT_TABLE table=default.Table_46b8725e01684c90bcbbde7b5a9f4e0e [id=39aeda29815d474196136813b5d811e7]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:33.286986 1717 leader_election.cc:343] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:33.347256 1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 60303.2us
I20260501 13:57:33.311707 1763 tablet_service.cc:1511] Processing CreateTablet for tablet e2ac522f2e4c4ab48a9b7d211c82ed9e (DEFAULT_TABLE table=default.Table_e69ec1df728e4af78640327a1b36ad4b [id=34b2db06c0a84cecbc10ec705caa4656]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.302057 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 0c736959df334aee857dbaae0f7a1e1a (DEFAULT_TABLE table=default.Table_56892e9ecd4b4b0b9133bc936b37c1c4 [id=09422f53fb424bdbaa00282b1775947d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.352487 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38c3eb388d474d718554775d65f58d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.357932 2097 raft_consensus.cc:493] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:33.358409 2097 raft_consensus.cc:515] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.361701 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c736959df334aee857dbaae0f7a1e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.362382 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c736959df334aee857dbaae0f7a1e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.363505 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.364324 1899 raft_consensus.cc:2243] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:33.372855 1899 pending_rounds.cc:70] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: Trying to abort 1 pending ops.
I20260501 13:57:33.327392 1883 tablet_bootstrap.cc:492] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:33.373994 1883 ts_tablet_manager.cc:1403] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.435s user 0.025s sys 0.032s
I20260501 13:57:33.374818 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 921e5c3eaaa44c62abbc90f560049614
I20260501 13:57:33.376839 1124 catalog_manager.cc:6741] ReplaceTablet: tablet 921e5c3eaaa44c62abbc90f560049614 deleted and replaced by tablet 68cb7bd1a5aa4419b9c5176a287fe885
I20260501 13:57:33.381121 1128 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 2ff05404061447199d936d005afda653
I20260501 13:57:33.383771 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ff05404061447199d936d005afda653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:33.387555 1126 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_a685edff90a04b37ba8dafcaea3d7f55 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 2 got 0
I20260501 13:57:33.391806 1131 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 9a0d04810f2d465ebe2964797ea30538
I20260501 13:57:33.389626 1882 tablet_bootstrap.cc:654] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:33.398340 1123 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 1ba63c153ef04db891cae46a71439455
I20260501 13:57:33.399905 1123 catalog_manager.cc:6741] ReplaceTablet: tablet 1ba63c153ef04db891cae46a71439455 deleted and replaced by tablet 19c1e0c30dce488baed9e046d90f0ba4
I20260501 13:57:33.279736 2092 tablet.cc:1724] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:33.208324 2047 ts_tablet_manager.cc:1929] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:33.377383 1458 tablet_service.cc:1558] Processing DeleteTablet for tablet 921e5c3eaaa44c62abbc90f560049614 with delete_type TABLET_DATA_DELETED (replaced by tablet 68cb7bd1a5aa4419b9c5176a287fe885) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:33.373139 1899 pending_rounds.cc:77] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7281229221471621120 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "45907da8b56f4fe887253c77ae12f9a3" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_4275f119eec547c28a46b8d5f4faa0f8" dest_uuid: "993b9ba18d8547f8af43167e727a6a24" new_extra_config { } }
I20260501 13:57:33.420254 1899 raft_consensus.cc:2272] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:33.420559 2097 leader_election.cc:290] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:33.420912 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 921e5c3eaaa44c62abbc90f560049614 with delete_type TABLET_DATA_DELETED (replaced by tablet 68cb7bd1a5aa4419b9c5176a287fe885) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:33.421818 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ac522f2e4c4ab48a9b7d211c82ed9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.435175 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_a3c73d27382042638126d9be32b33b96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:33.436929 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3c73d27382042638126d9be32b33b96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:33.440470 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_6e97f9768037452593b16b51f09a18ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:33.441956 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e97f9768037452593b16b51f09a18ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:33.419092 2047 log.cc:1199] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/89e2c22052d04fe1a21bcb3323385111
I20260501 13:57:33.445310 2098 tablet.cc:1719] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: Skipping requested alter to schema version 2, tablet already version 2
I20260501 13:57:33.449018 1881 tablet_bootstrap.cc:654] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:33.457638 1458 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ff05404061447199d936d005afda653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:33.458841 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a0d04810f2d465ebe2964797ea30538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:33.459446 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ba63c153ef04db891cae46a71439455 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c1e0c30dce488baed9e046d90f0ba4) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:33.460037 1931 raft_consensus.cc:493] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:33.460493 1931 raft_consensus.cc:515] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.462745 1127 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba] alter complete (version 1)
I20260501 13:57:33.467742 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c789ca9e3c748d5bc79f4e1b0ea8fd1" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:33.454921 1418 consensus_peers.cc:597] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 -> Peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Couldn't send request to peer 0ed1f9a545474e76a661c32bf11a7fb2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:33.473588 1899 ts_tablet_manager.cc:1916] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:33.479148 2047 ts_tablet_manager.cc:1950] T 89e2c22052d04fe1a21bcb3323385111 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:33.462332 1931 leader_election.cc:290] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:33.511844 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 76) took 1704 ms. Trace:
I20260501 13:57:33.512279 2047 rpcz_store.cc:276] 0501 13:57:31.807124 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:31.852695 (+ 45571us) service_pool.cc:224] Handling call
0501 13:57:31.950561 (+ 97866us) ts_tablet_manager.cc:1151] Deleting tablet 89e2c22052d04fe1a21bcb3323385111
0501 13:57:32.132943 (+182382us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b5800000098. stack: 00007f9e73a05c83 00007f9e73a05b80 00007f9e73576574 000000000050e094 000000000050e029 00007f9e73875941 00007f9e739fa6c1 00007f9e739facf2 00007f9e7c867469 00007f9e7baa7488 00007f9e7bb6e359 00007f9e7bb6e953 00007f9e7bb9f990 00007f9e7bb85efd 00007f9e7bb85ea9 00007f9e7bb85e39
0501 13:57:32.371162 (+238219us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:32.982663 (+611501us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:33.208192 (+225529us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:33.511758 (+303566us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":137309,"spinlock_wait_cycles":35280640,"tablet-delete.queue_time_us":16354}
I20260501 13:57:33.516865 2047 tablet_replica.cc:333] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:33.517961 2047 raft_consensus.cc:2243] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:33.518589 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 89e2c22052d04fe1a21bcb3323385111 (table default.Table_4b9a916d6a754ac7b38819b39d0b1671 [id=36e8cb1c80cc45e7b4c501608c374af9]) successfully deleted
I20260501 13:57:33.518968 2047 raft_consensus.cc:2272] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:33.529428 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_5d2b24ed68b54d58990aea3df63012ba" } new_table_name: "default.Table_785faee6678c46f1afa60872f1582c7f" modify_external_catalogs: true
I20260501 13:57:33.529788 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_c9c10d0b01f846849dcd931a75ff0d63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:33.531291 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9c10d0b01f846849dcd931a75ff0d63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:33.532545 1123 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_a3c73d27382042638126d9be32b33b96" } new_table_name: "default.Table_443326309dee4e12b1b00584159d6551" modify_external_catalogs: true
I20260501 13:57:33.545454 1881 tablet_bootstrap.cc:492] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:33.545883 1881 ts_tablet_manager.cc:1403] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.257s user 0.006s sys 0.006s
I20260501 13:57:33.552884 2047 ts_tablet_manager.cc:1916] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:33.553434 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9abb39c8b44d4b4c8b69751a4cdd6668 already in progress: creating tablet
W20260501 13:57:33.554085 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet adc47b3b2b084d33a12bd28cfad27382 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adc47b3b2b084d33a12bd28cfad27382 already in progress: creating tablet
I20260501 13:57:33.556128 2089 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:57818 (request call id 80) took 1441 ms. Trace:
I20260501 13:57:33.558552 1564 leader_election.cc:304] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:33.565119 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 921e5c3eaaa44c62abbc90f560049614 with delete_type TABLET_DATA_DELETED (replaced by tablet 68cb7bd1a5aa4419b9c5176a287fe885) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:33.556485 2089 rpcz_store.cc:276] 0501 13:57:32.129324 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.173936 (+ 44612us) service_pool.cc:224] Handling call
0501 13:57:33.556056 (+1382120us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:32.194305 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0501 13:57:32.198198 (+ 3893us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0501 13:57:32.198232 (+ 34us) alter_schema_op.cc:66] Acquired schema lock
0501 13:57:32.199063 (+ 831us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0501 13:57:32.199201 (+ 138us) alter_schema_op.cc:128] START. Timestamp: 1777643852199092
0501 13:57:32.199224 (+ 23us) op_driver.cc:348] REPLICATION: starting
0501 13:57:32.217310 (+ 18086us) log.cc:844] Serialized 173 byte log entry
0501 13:57:32.907667 (+690357us) op_driver.cc:464] REPLICATION: finished
0501 13:57:33.042370 (+134703us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0501 13:57:33.165084 (+122714us) log.cc:844] Serialized 5 byte log entry
0501 13:57:33.460316 (+295232us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:33.549474 (+ 89158us) log.cc:844] Serialized 14 byte log entry
0501 13:57:33.555916 (+ 6442us) alter_schema_op.cc:73] Released schema lock
0501 13:57:33.555948 (+ 32us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":134517,"mutex_wait_us":167236,"prepare.queue_time_us":179,"prepare.run_cpu_time_us":11310,"prepare.run_wall_time_us":358712,"raft.queue_time_us":182091,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":15,"replication_time_us":708345,"thread_start_us":287429,"threads_started":2,"wal-append.queue_time_us":377662}]]}
I20260501 13:57:33.566032 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 0c736959df334aee857dbaae0f7a1e1a (DEFAULT_TABLE table=default.Table_56892e9ecd4b4b0b9133bc936b37c1c4 [id=09422f53fb424bdbaa00282b1775947d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.560561 1881 raft_consensus.cc:359] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.566967 1881 raft_consensus.cc:385] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:33.567237 1881 raft_consensus.cc:740] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:33.567471 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c736959df334aee857dbaae0f7a1e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.572485 2107 raft_consensus.cc:2749] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:33.573336 1881 consensus_queue.cc:260] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.574942 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ff05404061447199d936d005afda653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:33.580348 1625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e15d66bf6147a08604daf2a9c53322" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:33.575634 1608 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ba63c153ef04db891cae46a71439455 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c1e0c30dce488baed9e046d90f0ba4) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:33.576229 1606 tablet_service.cc:1511] Processing CreateTablet for tablet e2ac522f2e4c4ab48a9b7d211c82ed9e (DEFAULT_TABLE table=default.Table_e69ec1df728e4af78640327a1b36ad4b [id=34b2db06c0a84cecbc10ec705caa4656]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.577117 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 (DEFAULT_TABLE table=default.Table_46b8725e01684c90bcbbde7b5a9f4e0e [id=39aeda29815d474196136813b5d811e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.582453 1624 raft_consensus.cc:3060] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:33.577554 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 38c3eb388d474d718554775d65f58d78 (DEFAULT_TABLE table=default.Table_8a77fe75f8044d75a97b45dd4d94216f [id=d6ca68cf0d954555b422b90e6502d6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.579150 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a0d04810f2d465ebe2964797ea30538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:33.585711 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_bc1aee595536420799f8a28881253c09" } modify_external_catalogs: true
I20260501 13:57:33.586305 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_bc1aee595536420799f8a28881253c09" } modify_external_catalogs: true
I20260501 13:57:33.582509 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ac522f2e4c4ab48a9b7d211c82ed9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.578293 1881 ts_tablet_manager.cc:1434] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.032s user 0.001s sys 0.005s
I20260501 13:57:33.587890 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.590874 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38c3eb388d474d718554775d65f58d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.595989 1128 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_e7f17dee34594eb3a711ffbf706d4e59" } modify_external_catalogs: true
I20260501 13:57:33.596514 1128 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_e7f17dee34594eb3a711ffbf706d4e59" } modify_external_catalogs: true
W20260501 13:57:33.606953 2028 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:57:33.607733 2028 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:57:33.608454 2028 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260501 13:57:33.628252 2047 ts_tablet_manager.cc:1929] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:57:33.628654 2047 log.cc:1199] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/45907da8b56f4fe887253c77ae12f9a3
I20260501 13:57:33.629406 2047 ts_tablet_manager.cc:1950] T 45907da8b56f4fe887253c77ae12f9a3 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:33.500001 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a0d04810f2d465ebe2964797ea30538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:33.640471 1883 raft_consensus.cc:359] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.641256 1883 raft_consensus.cc:385] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:33.641556 1883 raft_consensus.cc:740] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:33.642177 1883 consensus_queue.cc:260] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.655887 1883 ts_tablet_manager.cc:1434] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.282s user 0.010s sys 0.004s
W20260501 13:57:33.472532 1717 leader_election.cc:343] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:33.627409 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e15d66bf6147a08604daf2a9c53322" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
W20260501 13:57:33.667824 1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 195341us
I20260501 13:57:33.630115 2091 tablet.cc:1724] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:33.633085 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ba63c153ef04db891cae46a71439455 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c1e0c30dce488baed9e046d90f0ba4) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:33.671576 1881 tablet_bootstrap.cc:492] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
W20260501 13:57:33.672371 2028 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 13:57:33.672768 2028 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:57:33.673112 2028 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 13:57:33.673435 2028 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 13:57:33.679076 1418 leader_election.cc:304] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:33.681402 1931 raft_consensus.cc:2749] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:33.674183 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 82) took 1488 ms. Trace:
I20260501 13:57:33.682770 2047 rpcz_store.cc:276] 0501 13:57:32.186150 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.186252 (+ 102us) service_pool.cc:224] Handling call
0501 13:57:33.516694 (+1330442us) ts_tablet_manager.cc:1151] Deleting tablet 45907da8b56f4fe887253c77ae12f9a3
0501 13:57:33.516750 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:33.585011 (+ 68261us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:33.628118 (+ 43107us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:33.674123 (+ 46005us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329870}
I20260501 13:57:33.684111 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 84) took 1496 ms. Trace:
I20260501 13:57:33.684435 2047 rpcz_store.cc:276] 0501 13:57:32.188037 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.188207 (+ 170us) service_pool.cc:224] Handling call
0501 13:57:33.683751 (+1495544us) ts_tablet_manager.cc:1151] Deleting tablet eeae4369be954d9d9bad846484afb6ff
0501 13:57:33.683794 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:33.684056 (+ 262us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1493893}
I20260501 13:57:33.685575 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 88) took 1252 ms. Trace:
I20260501 13:57:33.685882 2047 rpcz_store.cc:276] 0501 13:57:32.432908 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.433020 (+ 112us) service_pool.cc:224] Handling call
0501 13:57:33.685257 (+1252237us) ts_tablet_manager.cc:1151] Deleting tablet b98187c7843e42b09d55c851117c4ac8
0501 13:57:33.685297 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:33.685528 (+ 231us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1251726}
I20260501 13:57:33.686654 1901 consensus_queue.cc:1048] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:33.698817 1899 ts_tablet_manager.cc:1929] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:57:33.699260 1899 log.cc:1199] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/45907da8b56f4fe887253c77ae12f9a3
I20260501 13:57:33.699970 1899 ts_tablet_manager.cc:1950] T 45907da8b56f4fe887253c77ae12f9a3 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:33.709239 1881 tablet_bootstrap.cc:654] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:33.710647 1883 tablet_bootstrap.cc:492] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:33.719045 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 89) took 1268 ms. Trace:
I20260501 13:57:33.719419 2047 rpcz_store.cc:276] 0501 13:57:32.450819 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.450922 (+ 103us) service_pool.cc:224] Handling call
0501 13:57:33.718647 (+1267725us) ts_tablet_manager.cc:1151] Deleting tablet b507a4d4c9c54bc68d92d9286794a264
0501 13:57:33.718696 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:33.718985 (+ 289us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266818}
I20260501 13:57:33.719578 2109 tablet.cc:1724] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:33.721473 2047 tablet_replica.cc:333] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:33.722265 2047 raft_consensus.cc:2243] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:33.725917 2047 raft_consensus.cc:2272] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:33.730059 2047 ts_tablet_manager.cc:1916] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:33.744041 2028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.16.190:45953,127.1.16.189:45897,127.1.16.188:38815
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.188:38815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.16.188
--webserver_port=41671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:57:33.751643 1630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "104708eb7fc0412dad0bd7e14eee9d13" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:33.757062 2028 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:57:33.759543 2028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260501 13:57:33.767149 1883 tablet_bootstrap.cc:654] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:33.788646 1881 tablet_bootstrap.cc:492] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:33.789120 1881 ts_tablet_manager.cc:1403] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.118s user 0.005s sys 0.008s
I20260501 13:57:33.792236 1881 raft_consensus.cc:359] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.792883 1881 raft_consensus.cc:385] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:33.793145 1881 raft_consensus.cc:740] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:33.806991 1881 consensus_queue.cc:260] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.810075 1883 tablet_bootstrap.cc:492] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:33.810516 1883 ts_tablet_manager.cc:1403] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.100s user 0.006s sys 0.010s
I20260501 13:57:33.813089 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 87) took 1833 ms. Trace:
I20260501 13:57:33.813678 1883 raft_consensus.cc:359] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.814347 1883 raft_consensus.cc:385] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:33.814644 1883 raft_consensus.cc:740] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:33.813475 1899 rpcz_store.cc:276] 0501 13:57:31.979802 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:31.979906 (+ 104us) service_pool.cc:224] Handling call
0501 13:57:33.271499 (+1291593us) ts_tablet_manager.cc:1151] Deleting tablet 45907da8b56f4fe887253c77ae12f9a3
0501 13:57:33.271538 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:33.636597 (+365059us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:33.698689 (+ 62092us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:33.813031 (+114342us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2,"tablet-delete.queue_time_us":1290906}
I20260501 13:57:33.815258 1883 consensus_queue.cc:260] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.817211 1883 ts_tablet_manager.cc:1434] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260501 13:57:33.816486 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 88) took 1715 ms. Trace:
I20260501 13:57:33.820729 1883 tablet_bootstrap.cc:492] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:33.819372 1899 rpcz_store.cc:276] 0501 13:57:32.100941 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.111175 (+ 10234us) service_pool.cc:224] Handling call
0501 13:57:33.816141 (+1704966us) ts_tablet_manager.cc:1151] Deleting tablet eeae4369be954d9d9bad846484afb6ff
0501 13:57:33.816187 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:33.816433 (+ 246us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1704054}
I20260501 13:57:33.823145 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 93) took 1492 ms. Trace:
W20260501 13:57:33.827422 1116 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (table default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7f9c92846d4e4255a1c8ce3835a0a18d consensus state has no leader
I20260501 13:57:33.832064 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 45907da8b56f4fe887253c77ae12f9a3 (table default.Table_4275f119eec547c28a46b8d5f4faa0f8 [id=5cd8cae1c55f41aa809960d1cecddec3]) successfully deleted
W20260501 13:57:33.833115 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet eeae4369be954d9d9bad846484afb6ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeae4369be954d9d9bad846484afb6ff already in progress: creating tablet
I20260501 13:57:33.835191 1882 tablet_bootstrap.cc:492] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:33.835633 1882 ts_tablet_manager.cc:1403] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.777s user 0.013s sys 0.007s
I20260501 13:57:33.836987 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 45907da8b56f4fe887253c77ae12f9a3 (table default.Table_4275f119eec547c28a46b8d5f4faa0f8 [id=5cd8cae1c55f41aa809960d1cecddec3]) successfully deleted
W20260501 13:57:33.837841 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet eeae4369be954d9d9bad846484afb6ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeae4369be954d9d9bad846484afb6ff already in progress: creating tablet
W20260501 13:57:33.840157 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet b98187c7843e42b09d55c851117c4ac8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b98187c7843e42b09d55c851117c4ac8 already in progress: creating tablet
W20260501 13:57:33.840703 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 282386us
I20260501 13:57:33.848917 2093 tablet_replica.cc:333] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:33.847992 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7a4f0da973284ac49c7dd7ff3e99e09f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:33.849797 2093 raft_consensus.cc:2243] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Raft consensus shutting down.
W20260501 13:57:33.840873 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet b507a4d4c9c54bc68d92d9286794a264 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b507a4d4c9c54bc68d92d9286794a264 already in progress: creating tablet
W20260501 13:57:33.851095 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet adc47b3b2b084d33a12bd28cfad27382 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adc47b3b2b084d33a12bd28cfad27382 already in progress: creating tablet
W20260501 13:57:33.851814 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9abb39c8b44d4b4c8b69751a4cdd6668 already in progress: creating tablet
W20260501 13:57:33.851851 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a4f0da973284ac49c7dd7ff3e99e09f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:33.853830 1124 master_service.cc:954] ReplaceTablet: received request to replace tablet 863a10cfd14c4694b7c6141c0851167a from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:33.855456 1457 tablet_service.cc:1558] Processing DeleteTablet for tablet f2b59b51d81249e0ad90edba16803285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:33.856855 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_2e85e5364e5f4e05b0909a6cb58843c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:33.870237 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e85e5364e5f4e05b0909a6cb58843c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:33.871377 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_9f356943ea5a469c8eea733dc8231c36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:33.872896 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f356943ea5a469c8eea733dc8231c36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:33.875830 1881 ts_tablet_manager.cc:1434] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.086s user 0.003s sys 0.004s
I20260501 13:57:33.876755 1881 tablet_bootstrap.cc:492] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:33.834326 1628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c789ca9e3c748d5bc79f4e1b0ea8fd1" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:33.879299 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33a355897309420c97b1257afaefe8d1 consensus state has no leader
I20260501 13:57:33.880054 1131 master_service.cc:954] ReplaceTablet: received request to replace tablet e41b43329e054442bc8697cc28161954 from {username='slave'} at 127.0.0.1:41700
W20260501 13:57:33.882601 1123 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 967658b1be204c638fc34983293ef5ab has no consensus state
I20260501 13:57:33.891872 1882 raft_consensus.cc:359] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.892730 1882 raft_consensus.cc:385] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:33.893224 1882 raft_consensus.cc:740] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:33.901355 2093 raft_consensus.cc:2272] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:33.905143 2047 ts_tablet_manager.cc:1929] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:33.905866 2047 log.cc:1199] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/bf96cb4696704478a2064d6ce785d49d
W20260501 13:57:33.846839 1719 leader_election.cc:343] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:33.867516 1883 tablet_bootstrap.cc:654] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:33.918582 1719 leader_election.cc:304] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:33.919278 2097 raft_consensus.cc:2749] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:33.920096 1719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 73311.1us
W20260501 13:57:33.920869 1719 consensus_peers.cc:597] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:33.920902 1931 raft_consensus.cc:493] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 13:57:33.921571 1719 leader_election.cc:343] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:33.921362 1931 raft_consensus.cc:515] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.922055 1719 leader_election.cc:304] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:33.922252 1881 tablet_bootstrap.cc:654] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:33.922761 2097 raft_consensus.cc:2749] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:33.923913 1899 rpcz_store.cc:276] 0501 13:57:32.330266 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.330390 (+ 124us) service_pool.cc:224] Handling call
0501 13:57:33.822819 (+1492429us) ts_tablet_manager.cc:1151] Deleting tablet b98187c7843e42b09d55c851117c4ac8
0501 13:57:33.822859 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:33.823092 (+ 233us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491460}
W20260501 13:57:33.925052 1418 leader_election.cc:343] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: f2b59b51d81249e0ad90edba16803285
I20260501 13:57:33.925398 1899 tablet_replica.cc:333] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:33.926122 1931 leader_election.cc:290] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:33.926270 1899 raft_consensus.cc:2243] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:33.926853 1899 raft_consensus.cc:2272] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:33.927089 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet b98187c7843e42b09d55c851117c4ac8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b98187c7843e42b09d55c851117c4ac8 already in progress: creating tablet
I20260501 13:57:33.893921 1882 consensus_queue.cc:260] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:33.928660 2047 ts_tablet_manager.cc:1950] T bf96cb4696704478a2064d6ce785d49d P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
W20260501 13:57:33.943568 2118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:57:33.951880 2047 tablet_replica.cc:333] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:33.953392 2047 raft_consensus.cc:2243] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:33.933646 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet bf96cb4696704478a2064d6ce785d49d (table default.Table_4c1ccc461d624514a37044a124d8ad6b [id=314708984ce445e2a57e799bc77d5c6f]) successfully deleted
I20260501 13:57:33.960958 1763 tablet_service.cc:1511] Processing CreateTablet for tablet e8efee17e2c34caf98c0ede01d2a943a (DEFAULT_TABLE table=default.Table_89c5b57152224983bcb4b5ef5acd7af4 [id=cf2193b6ac264ef290b4bfb40970f215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.961488 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 7d399de014d74cab8df14f44787ca2f7 (DEFAULT_TABLE table=default.Table_aa47824eda84455d9d52df61514a253e [id=81117c514ffd4ab6a8d4dde20d7559e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.961921 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8efee17e2c34caf98c0ede01d2a943a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.962354 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d399de014d74cab8df14f44787ca2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.966265 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 7d399de014d74cab8df14f44787ca2f7 (DEFAULT_TABLE table=default.Table_aa47824eda84455d9d52df61514a253e [id=81117c514ffd4ab6a8d4dde20d7559e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.986845 1899 ts_tablet_manager.cc:1916] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:33.987169 2047 raft_consensus.cc:2272] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:33.987514 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d399de014d74cab8df14f44787ca2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.987733 1127 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet c5297eb8aace46078cc3fce289fdd389 (table default.Table_eb039fccfc7d444baf4e6cdddc6be5cd [id=37089fa463db4a608b09504153e2e32a]). Expected version 1 got 0
I20260501 13:57:33.989145 2047 ts_tablet_manager.cc:1916] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:33.966815 1457 tablet_service.cc:1511] Processing CreateTablet for tablet e8efee17e2c34caf98c0ede01d2a943a (DEFAULT_TABLE table=default.Table_89c5b57152224983bcb4b5ef5acd7af4 [id=cf2193b6ac264ef290b4bfb40970f215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.992964 1457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8efee17e2c34caf98c0ede01d2a943a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.970389 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 9169851f1358460d99a48cb6067ab3e3 (DEFAULT_TABLE table=default.Table_51bf305a64b9456c8cb5a9c56799f4cb [id=77bda28326844d3bae26bbedf30f8e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.995258 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9169851f1358460d99a48cb6067ab3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.982162 1460 tablet_service.cc:1511] Processing CreateTablet for tablet 69626ed095bd4df7859563c176dafffc (DEFAULT_TABLE table=default.Table_5e1da2dc1631431c9ced9ebb18d903d8 [id=63f1fe234b1143869f21b76118b58234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:33.999356 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69626ed095bd4df7859563c176dafffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:33.999809 1758 tablet_service.cc:1511] Processing CreateTablet for tablet 9169851f1358460d99a48cb6067ab3e3 (DEFAULT_TABLE table=default.Table_51bf305a64b9456c8cb5a9c56799f4cb [id=77bda28326844d3bae26bbedf30f8e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.000715 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9169851f1358460d99a48cb6067ab3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.007786 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 69626ed095bd4df7859563c176dafffc (DEFAULT_TABLE table=default.Table_5e1da2dc1631431c9ced9ebb18d903d8 [id=63f1fe234b1143869f21b76118b58234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.008787 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69626ed095bd4df7859563c176dafffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.011909 1881 tablet_bootstrap.cc:492] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:34.012318 1881 ts_tablet_manager.cc:1403] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.136s user 0.008s sys 0.000s
W20260501 13:57:34.026573 1419 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260501 13:57:34.030736 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 199016us
I20260501 13:57:34.034400 1881 raft_consensus.cc:359] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.049077 1899 ts_tablet_manager.cc:1929] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260501 13:57:34.052225 1119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.188:38815, user_credentials={real_user=slave}} blocked reactor thread for 44421us
I20260501 13:57:34.062971 2093 ts_tablet_manager.cc:1916] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:34.063766 1129 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet c5297eb8aace46078cc3fce289fdd389 (table default.Table_eb039fccfc7d444baf4e6cdddc6be5cd [id=37089fa463db4a608b09504153e2e32a]). Expected version 1 got 0
I20260501 13:57:34.066654 1899 log.cc:1199] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/b507a4d4c9c54bc68d92d9286794a264
I20260501 13:57:34.067406 1899 ts_tablet_manager.cc:1950] T b507a4d4c9c54bc68d92d9286794a264 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:34.069088 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 7d399de014d74cab8df14f44787ca2f7 (DEFAULT_TABLE table=default.Table_aa47824eda84455d9d52df61514a253e [id=81117c514ffd4ab6a8d4dde20d7559e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.070036 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d399de014d74cab8df14f44787ca2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.070298 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 9169851f1358460d99a48cb6067ab3e3 (DEFAULT_TABLE table=default.Table_51bf305a64b9456c8cb5a9c56799f4cb [id=77bda28326844d3bae26bbedf30f8e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.071169 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9169851f1358460d99a48cb6067ab3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.071058 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 69626ed095bd4df7859563c176dafffc (DEFAULT_TABLE table=default.Table_5e1da2dc1631431c9ced9ebb18d903d8 [id=63f1fe234b1143869f21b76118b58234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.071902 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69626ed095bd4df7859563c176dafffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.072672 1607 tablet_service.cc:1511] Processing CreateTablet for tablet e8efee17e2c34caf98c0ede01d2a943a (DEFAULT_TABLE table=default.Table_89c5b57152224983bcb4b5ef5acd7af4 [id=cf2193b6ac264ef290b4bfb40970f215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.073493 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8efee17e2c34caf98c0ede01d2a943a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.076041 1128 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb
I20260501 13:57:34.076906 1610 tablet_service.cc:1511] Processing CreateTablet for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb (DEFAULT_TABLE table=default.Table_e7f17dee34594eb3a711ffbf706d4e59 [id=8570b73bebda4640a91c1bc894942e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.077664 2109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:60694 (request call id 107) took 1205 ms. Trace:
I20260501 13:57:34.077965 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.080341 1882 ts_tablet_manager.cc:1434] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.244s user 0.005s sys 0.002s
I20260501 13:57:34.078075 2109 rpcz_store.cc:276] 0501 13:57:32.871946 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.875433 (+ 3487us) service_pool.cc:224] Handling call
0501 13:57:34.077569 (+1202136us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:32.876414 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0501 13:57:32.876740 (+ 326us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0501 13:57:32.876768 (+ 28us) alter_schema_op.cc:66] Acquired schema lock
0501 13:57:32.876810 (+ 42us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0501 13:57:32.876965 (+ 155us) alter_schema_op.cc:128] START. Timestamp: 1777643852876879
0501 13:57:32.876985 (+ 20us) op_driver.cc:348] REPLICATION: starting
0501 13:57:32.877424 (+ 439us) log.cc:844] Serialized 173 byte log entry
0501 13:57:33.581543 (+704119us) op_driver.cc:464] REPLICATION: finished
0501 13:57:33.719329 (+137786us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0501 13:57:33.720917 (+ 1588us) log.cc:844] Serialized 5 byte log entry
0501 13:57:34.061143 (+340226us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.076328 (+ 15185us) log.cc:844] Serialized 14 byte log entry
0501 13:57:34.077419 (+ 1091us) alter_schema_op.cc:73] Released schema lock
0501 13:57:34.077448 (+ 29us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":137553,"mutex_wait_us":7635,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":1246,"prepare.run_wall_time_us":1247,"raft.queue_time_us":955,"raft.run_cpu_time_us":710,"raft.run_wall_time_us":909,"replication_time_us":704300,"spinlock_wait_cycles":717056,"thread_start_us":141718,"threads_started":1}]]}
I20260501 13:57:34.080744 1881 raft_consensus.cc:385] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:34.057497 1459 tablet_service.cc:1511] Processing CreateTablet for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb (DEFAULT_TABLE table=default.Table_e7f17dee34594eb3a711ffbf706d4e59 [id=8570b73bebda4640a91c1bc894942e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.068326 1482 raft_consensus.cc:3060] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:34.081254 1882 tablet_bootstrap.cc:492] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:34.081593 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.079599 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:34.087563 1608 tablet_service.cc:1558] Processing DeleteTablet for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:34.075238 1758 tablet_service.cc:1511] Processing CreateTablet for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb (DEFAULT_TABLE table=default.Table_e7f17dee34594eb3a711ffbf706d4e59 [id=8570b73bebda4640a91c1bc894942e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.095312 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.100867 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 863a10cfd14c4694b7c6141c0851167a
I20260501 13:57:34.102156 1124 catalog_manager.cc:6741] ReplaceTablet: tablet 863a10cfd14c4694b7c6141c0851167a deleted and replaced by tablet 6f6cd353ca044465a6692656412e7d82
I20260501 13:57:34.102347 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_b6780bd7a18f45d8b8ca54038933768f" } new_table_name: "default.Table_4440a2fc30004b389edb91265721ba86" modify_external_catalogs: true
I20260501 13:57:34.106109 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c6e79351ac8041739c99caa55b4af0ff" } new_table_name: "default.Table_ecd46a159d2448f9ba09fc99e14a22a3" modify_external_catalogs: true
I20260501 13:57:34.106479 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_eeb1c65f0d0448cba002f35e5a5aeb5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:34.107887 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eeb1c65f0d0448cba002f35e5a5aeb5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:34.110443 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet 863a10cfd14c4694b7c6141c0851167a with delete_type TABLET_DATA_DELETED (replaced by tablet 6f6cd353ca044465a6692656412e7d82) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:34.081012 1881 raft_consensus.cc:740] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:34.135548 1881 consensus_queue.cc:260] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.075798 1883 tablet_bootstrap.cc:492] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:34.080111 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:34.137133 1883 ts_tablet_manager.cc:1403] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.317s user 0.016s sys 0.014s
I20260501 13:57:34.139118 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 863a10cfd14c4694b7c6141c0851167a with delete_type TABLET_DATA_DELETED (replaced by tablet 6f6cd353ca044465a6692656412e7d82) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:34.142796 1881 ts_tablet_manager.cc:1434] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.130s user 0.008s sys 0.000s
I20260501 13:57:34.143553 2136 tablet.cc:1724] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:34.145959 1882 tablet_bootstrap.cc:654] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:34.154016 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet 863a10cfd14c4694b7c6141c0851167a with delete_type TABLET_DATA_DELETED (replaced by tablet 6f6cd353ca044465a6692656412e7d82) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:34.159684 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_fb6bb57c871e4855aba95f3b430a3fc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:34.161276 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb6bb57c871e4855aba95f3b430a3fc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:34.173377 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 94) took 1832 ms. Trace:
I20260501 13:57:34.173812 1899 rpcz_store.cc:276] 0501 13:57:32.341381 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.341515 (+ 134us) service_pool.cc:224] Handling call
0501 13:57:33.925238 (+1583723us) ts_tablet_manager.cc:1151] Deleting tablet b507a4d4c9c54bc68d92d9286794a264
0501 13:57:33.925286 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:34.000078 (+ 74792us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.048946 (+ 48868us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.173314 (+124368us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34560,"tablet-delete.queue_time_us":1583204}
I20260501 13:57:34.176258 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet b507a4d4c9c54bc68d92d9286794a264 (table default.Table_83779be1d30c4839999b9217ce2e4a20 [id=36cedf51b8a8402eb9488172828f7eb8]) successfully deleted
I20260501 13:57:34.183748 1127 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_a685edff90a04b37ba8dafcaea3d7f55 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 2 got 0
I20260501 13:57:34.186264 1881 tablet_bootstrap.cc:492] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:34.194854 1899 tablet_replica.cc:333] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:34.195173 1883 raft_consensus.cc:359] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.195847 1883 raft_consensus.cc:385] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:34.196101 1883 raft_consensus.cc:740] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:34.196667 1883 consensus_queue.cc:260] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.214592 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet a9672aab85c64d548a4f6583a1e9af20
I20260501 13:57:34.216984 2109 tablet.cc:1719] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: Skipping requested alter to schema version 2, tablet already version 2
I20260501 13:57:34.221750 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet a9672aab85c64d548a4f6583a1e9af20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:34.224784 1758 tablet_service.cc:1558] Processing DeleteTablet for tablet a9672aab85c64d548a4f6583a1e9af20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:34.225606 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b59b51d81249e0ad90edba16803285" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:34.228922 1457 tablet_service.cc:1558] Processing DeleteTablet for tablet a9672aab85c64d548a4f6583a1e9af20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:34.232228 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_a27ca155bb494a99bc23aefa4e6823f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:34.232213 2109 tablet.cc:1724] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 13:57:34.233683 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a27ca155bb494a99bc23aefa4e6823f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:34.241566 1131 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet b8d0f6d1f38b428291dba22015ac6959 (table default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba]). Expected version 1 got 0
W20260501 13:57:34.243882 1421 leader_election.cc:343] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:34.244313 1421 leader_election.cc:304] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:34.245788 1931 raft_consensus.cc:2749] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:34.249811 2093 ts_tablet_manager.cc:1929] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260501 13:57:34.267635 2093 log.cc:1199] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/921e5c3eaaa44c62abbc90f560049614
I20260501 13:57:34.261032 1882 tablet_bootstrap.cc:492] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:34.268373 1882 ts_tablet_manager.cc:1403] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.187s user 0.015s sys 0.000s
I20260501 13:57:34.269404 2093 ts_tablet_manager.cc:1950] T 921e5c3eaaa44c62abbc90f560049614 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:34.270061 2097 raft_consensus.cc:493] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:34.271198 2097 raft_consensus.cc:515] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.273010 2097 leader_election.cc:290] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:34.273559 1881 tablet_bootstrap.cc:654] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:34.200222 1899 raft_consensus.cc:2243] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:34.276032 1899 raft_consensus.cc:2272] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:34.276466 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "920d7f9748ae403d90573d612feeba59" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:34.277139 1125 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_a685edff90a04b37ba8dafcaea3d7f55 [id=db653ae15d0b4dada66cc4a7b2ac94bd] alter complete (version 2)
I20260501 13:57:34.277686 1899 ts_tablet_manager.cc:1916] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:34.278419 1719 leader_election.cc:343] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:34.283191 1883 ts_tablet_manager.cc:1434] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.146s user 0.006s sys 0.000s
I20260501 13:57:34.282996 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_cc6986b178ed47349a3dfda0ef76ce70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:34.284112 1883 tablet_bootstrap.cc:492] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:34.285569 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc6986b178ed47349a3dfda0ef76ce70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:34.291117 1882 raft_consensus.cc:359] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.291317 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_cf80c30d6b014d56aab7e294515801d5" } modify_external_catalogs: true
I20260501 13:57:34.291764 1882 raft_consensus.cc:385] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:34.291887 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_cf80c30d6b014d56aab7e294515801d5" } modify_external_catalogs: true
I20260501 13:57:34.292018 1882 raft_consensus.cc:740] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:34.296670 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47]) successfully deleted
W20260501 13:57:34.297662 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 2ff05404061447199d936d005afda653 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ff05404061447199d936d005afda653 already in progress: creating tablet
I20260501 13:57:34.300370 1758 tablet_service.cc:1558] Processing DeleteTablet for tablet b507a4d4c9c54bc68d92d9286794a264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:34.302575 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "920d7f9748ae403d90573d612feeba59" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:34.303788 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 1ba63c153ef04db891cae46a71439455 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ba63c153ef04db891cae46a71439455 already in progress: creating tablet
I20260501 13:57:34.304733 1882 consensus_queue.cc:260] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:34.306419 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 9a0d04810f2d465ebe2964797ea30538 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a0d04810f2d465ebe2964797ea30538 already in progress: creating tablet
W20260501 13:57:34.306833 1717 leader_election.cc:343] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:34.307250 1717 leader_election.cc:304] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:34.307951 2097 raft_consensus.cc:2749] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:34.309643 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb already in progress: creating tablet
W20260501 13:57:34.313762 1421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 68413.3us
I20260501 13:57:34.319425 1882 ts_tablet_manager.cc:1434] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.051s user 0.005s sys 0.000s
I20260501 13:57:34.320168 1607 tablet_service.cc:1558] Processing DeleteTablet for tablet bf96cb4696704478a2064d6ce785d49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:34.322093 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 863a10cfd14c4694b7c6141c0851167a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 863a10cfd14c4694b7c6141c0851167a already in progress: creating tablet
I20260501 13:57:34.323848 1882 tablet_bootstrap.cc:492] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:34.328938 1883 tablet_bootstrap.cc:654] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:34.346575 1883 tablet_bootstrap.cc:492] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:34.347118 1883 ts_tablet_manager.cc:1403] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.009s
I20260501 13:57:34.349606 1607 tablet_service.cc:1558] Processing DeleteTablet for tablet 89e2c22052d04fe1a21bcb3323385111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:34.351343 1883 raft_consensus.cc:359] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.352108 1883 raft_consensus.cc:385] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:34.352402 1883 raft_consensus.cc:740] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:34.353104 1883 consensus_queue.cc:260] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.354800 2093 tablet_replica.cc:333] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:34.355803 2093 raft_consensus.cc:2243] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
W20260501 13:57:34.362085 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet a9672aab85c64d548a4f6583a1e9af20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9672aab85c64d548a4f6583a1e9af20 already in progress: creating tablet
W20260501 13:57:34.363878 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c33a1f423bdb44f5843daffe8f73e74a consensus state has no leader
I20260501 13:57:34.356256 2093 raft_consensus.cc:2272] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:34.368021 2093 ts_tablet_manager.cc:1916] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:34.370002 1883 ts_tablet_manager.cc:1434] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20260501 13:57:34.370976 1883 tablet_bootstrap.cc:492] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:34.372748 1128 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4265d9ab3734a8b97d745dc02c6b845 has no consensus state
I20260501 13:57:34.379293 1883 tablet_bootstrap.cc:654] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:34.390795 2136 tablet.cc:1724] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:34.402788 2047 ts_tablet_manager.cc:1929] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260501 13:57:34.403187 2047 log.cc:1199] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/921e5c3eaaa44c62abbc90f560049614
I20260501 13:57:34.403870 2047 ts_tablet_manager.cc:1950] T 921e5c3eaaa44c62abbc90f560049614 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:34.404609 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 105) took 1027 ms. Trace:
I20260501 13:57:34.404923 2047 rpcz_store.cc:276] 0501 13:57:33.377092 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:33.377239 (+ 147us) service_pool.cc:224] Handling call
0501 13:57:33.951696 (+574457us) ts_tablet_manager.cc:1151] Deleting tablet 921e5c3eaaa44c62abbc90f560049614
0501 13:57:33.951752 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:34.237984 (+286232us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.402662 (+164678us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.404555 (+ 1893us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":532363}
I20260501 13:57:34.418712 1881 tablet_bootstrap.cc:492] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:34.419121 1881 ts_tablet_manager.cc:1403] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.233s user 0.006s sys 0.006s
I20260501 13:57:34.418982 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_9f356943ea5a469c8eea733dc8231c36" } modify_external_catalogs: true
I20260501 13:57:34.419468 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_9f356943ea5a469c8eea733dc8231c36" } modify_external_catalogs: true
I20260501 13:57:34.421315 1881 raft_consensus.cc:359] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.421887 1881 raft_consensus.cc:385] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:34.422132 1881 raft_consensus.cc:740] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:34.423758 1883 tablet_bootstrap.cc:492] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:34.424139 1883 ts_tablet_manager.cc:1403] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.053s user 0.015s sys 0.000s
I20260501 13:57:34.406741 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47]) successfully deleted
I20260501 13:57:34.426558 1883 raft_consensus.cc:359] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.427181 1883 raft_consensus.cc:385] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:34.427438 1883 raft_consensus.cc:740] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:34.428098 1899 ts_tablet_manager.cc:1929] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:34.429738 1899 log.cc:1199] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/104708eb7fc0412dad0bd7e14eee9d13
I20260501 13:57:34.428027 1883 consensus_queue.cc:260] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.433166 1883 ts_tablet_manager.cc:1434] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260501 13:57:34.434378 1881 consensus_queue.cc:260] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.434914 1899 ts_tablet_manager.cc:1950] T 104708eb7fc0412dad0bd7e14eee9d13 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:34.435405 1882 tablet_bootstrap.cc:654] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:34.438244 1883 tablet_bootstrap.cc:492] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:34.442672 1124 master_service.cc:954] ReplaceTablet: received request to replace tablet 4a563bccc7b046658ed133109f676528 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:34.452577 1881 ts_tablet_manager.cc:1434] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260501 13:57:34.437654 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 95) took 1999 ms. Trace:
I20260501 13:57:34.455065 1899 rpcz_store.cc:276] 0501 13:57:32.437787 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.437904 (+ 117us) service_pool.cc:224] Handling call
0501 13:57:34.194684 (+1756780us) ts_tablet_manager.cc:1151] Deleting tablet 104708eb7fc0412dad0bd7e14eee9d13
0501 13:57:34.194738 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:34.325987 (+131249us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.427986 (+101999us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.437571 (+ 9585us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1756285}
I20260501 13:57:34.456219 1899 tablet_replica.cc:333] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:34.456353 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 104708eb7fc0412dad0bd7e14eee9d13 (table default.Table_4a57dcc1c3194af793a3a0367e766cc6 [id=a921f986b8ac4a948ccb297e1084fccb]) successfully deleted
I20260501 13:57:34.456967 1899 raft_consensus.cc:2243] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:34.457381 1899 raft_consensus.cc:2272] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:34.465649 1130 master_service.cc:954] ReplaceTablet: received request to replace tablet ec4bd4cb1e70472aa09fc93afc3b1501 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:34.472273 1881 tablet_bootstrap.cc:492] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:34.472661 1882 tablet_bootstrap.cc:492] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:34.473055 1882 ts_tablet_manager.cc:1403] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.149s user 0.006s sys 0.003s
I20260501 13:57:34.462826 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 106) took 1019 ms. Trace:
I20260501 13:57:34.473667 2047 rpcz_store.cc:276] 0501 13:57:33.457360 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:33.457473 (+ 113us) service_pool.cc:224] Handling call
0501 13:57:34.406428 (+948955us) ts_tablet_manager.cc:1151] Deleting tablet 2ff05404061447199d936d005afda653
0501 13:57:34.455836 (+ 49408us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:34.462744 (+ 6908us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":948318}
I20260501 13:57:34.475225 1882 raft_consensus.cc:359] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:34.475544 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 2ff05404061447199d936d005afda653 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ff05404061447199d936d005afda653 already in progress: creating tablet
I20260501 13:57:34.476001 2097 raft_consensus.cc:493] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:34.476408 2097 raft_consensus.cc:515] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.479562 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1409fbe5846a4bb099f31fb2d2e41382" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:34.480836 1719 leader_election.cc:343] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:34.478312 2097 leader_election.cc:290] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:34.481868 1931 raft_consensus.cc:493] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:34.482843 1931 raft_consensus.cc:515] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.487773 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1409fbe5846a4bb099f31fb2d2e41382" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:34.488993 1717 leader_election.cc:343] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:34.489101 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 107) took 1031 ms. Trace:
I20260501 13:57:34.489391 1717 leader_election.cc:304] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:34.489460 2047 rpcz_store.cc:276] 0501 13:57:33.457761 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:33.458721 (+ 960us) service_pool.cc:224] Handling call
0501 13:57:34.476478 (+1017757us) ts_tablet_manager.cc:1151] Deleting tablet 9a0d04810f2d465ebe2964797ea30538
0501 13:57:34.487515 (+ 11037us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:34.489021 (+ 1506us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017226}
I20260501 13:57:34.489974 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5297eb8aace46078cc3fce289fdd389" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:34.490571 1783 raft_consensus.cc:2468] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:34.490844 1883 tablet_bootstrap.cc:654] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:34.490965 2097 raft_consensus.cc:2749] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:34.490960 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 9a0d04810f2d465ebe2964797ea30538 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a0d04810f2d465ebe2964797ea30538 already in progress: creating tablet
I20260501 13:57:34.491923 1418 leader_election.cc:304] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:34.492365 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 108) took 1034 ms. Trace:
I20260501 13:57:34.492596 2138 raft_consensus.cc:2804] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:34.492961 2138 raft_consensus.cc:493] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:34.493323 2138 raft_consensus.cc:3060] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:34.492710 2047 rpcz_store.cc:276] 0501 13:57:33.458089 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:33.459335 (+ 1246us) service_pool.cc:224] Handling call
0501 13:57:34.491534 (+1032199us) ts_tablet_manager.cc:1151] Deleting tablet 1ba63c153ef04db891cae46a71439455
0501 13:57:34.491963 (+ 429us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:34.492307 (+ 344us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1031682}
W20260501 13:57:34.494583 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 1ba63c153ef04db891cae46a71439455 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ba63c153ef04db891cae46a71439455 already in progress: creating tablet
I20260501 13:57:34.496724 2047 tablet_replica.cc:333] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:34.496975 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5297eb8aace46078cc3fce289fdd389" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:34.497319 1881 tablet_bootstrap.cc:654] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:34.497511 2047 raft_consensus.cc:2243] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:34.497993 2047 raft_consensus.cc:2272] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:34.498440 1421 leader_election.cc:343] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:34.499046 1931 leader_election.cc:290] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:34.499980 2047 ts_tablet_manager.cc:1916] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:34.475814 1882 raft_consensus.cc:385] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:34.506862 1882 raft_consensus.cc:740] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:34.511583 1882 consensus_queue.cc:260] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.518666 1899 ts_tablet_manager.cc:1916] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:34.518312 1882 ts_tablet_manager.cc:1434] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.045s user 0.006s sys 0.001s
I20260501 13:57:34.530828 1882 tablet_bootstrap.cc:492] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:34.542196 1883 tablet_bootstrap.cc:492] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:34.542671 1883 ts_tablet_manager.cc:1403] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.105s user 0.010s sys 0.001s
I20260501 13:57:34.552392 2138 raft_consensus.cc:515] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.554268 2138 leader_election.cc:290] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:34.555431 1883 raft_consensus.cc:359] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.555917 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5297eb8aace46078cc3fce289fdd389" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:34.556193 1883 raft_consensus.cc:385] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:34.556429 1883 raft_consensus.cc:740] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:34.557032 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5297eb8aace46078cc3fce289fdd389" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:34.556988 1883 consensus_queue.cc:260] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.557603 1783 raft_consensus.cc:3060] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:34.559262 1421 leader_election.cc:343] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:34.562649 2097 raft_consensus.cc:493] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:34.563112 2097 raft_consensus.cc:515] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.564877 2097 leader_election.cc:290] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:34.566408 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a57bc58e854e469897a04b2d434d06" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:34.567720 1719 leader_election.cc:343] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:34.568511 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_3d853a6bfdc945fd9fb7c23468905983" } new_table_name: "default.Table_d56638cc43fa42f1a0da405e69383a6a" modify_external_catalogs: true
I20260501 13:57:34.571179 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a57bc58e854e469897a04b2d434d06" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:34.572405 1717 leader_election.cc:343] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:34.572790 1717 leader_election.cc:304] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:34.573457 2097 raft_consensus.cc:2749] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:34.574622 1125 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_a685edff90a04b37ba8dafcaea3d7f55 [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 2 got 1
I20260501 13:57:34.575166 1125 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 6c2d5336bfec43dbb38ce7071834bca3 (table default.Table_66545ca4c3d540ada57273201e17f2d8 [id=697eaa78d5e541988cd48466c64c5915]). Expected version 1 got 0
I20260501 13:57:34.576762 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_d17e515649da406b814a3e991d8f0d41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:34.578227 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d17e515649da406b814a3e991d8f0d41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:34.578878 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_a685edff90a04b37ba8dafcaea3d7f55" } new_table_name: "default.Table_724820186f1a45fc99ca679fdc8cc36d" modify_external_catalogs: true
I20260501 13:57:34.584158 1883 ts_tablet_manager.cc:1434] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20260501 13:57:34.588171 1883 tablet_bootstrap.cc:492] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:34.592161 1881 tablet_bootstrap.cc:492] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:34.592617 1881 ts_tablet_manager.cc:1403] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.121s user 0.010s sys 0.002s
I20260501 13:57:34.594498 1882 tablet_bootstrap.cc:654] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:34.597505 1123 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 920d7f9748ae403d90573d612feeba59 (table default.Table_f0801819716e426f92844203c4a06bdc [id=068f5ac17d494d0b8639cbdc07a04597]). Expected version 2 got 0
I20260501 13:57:34.602907 1883 tablet_bootstrap.cc:654] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:34.607718 1783 raft_consensus.cc:2468] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:34.609010 1418 leader_election.cc:304] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:34.609820 2138 raft_consensus.cc:2804] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:34.610244 2138 raft_consensus.cc:697] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:34.611105 2138 consensus_queue.cc:237] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.619196 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_c3fddeb4be4642b7b940533315be33c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:34.620666 1899 ts_tablet_manager.cc:1929] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:34.621078 1899 log.cc:1199] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/2ee580bb9d3843f9bbeae8eef2c347ad
I20260501 13:57:34.621881 1899 ts_tablet_manager.cc:1950] T 2ee580bb9d3843f9bbeae8eef2c347ad P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:34.623407 2108 raft_consensus.cc:493] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:34.623165 1881 raft_consensus.cc:359] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.623801 1881 raft_consensus.cc:385] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:34.623742 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 106) took 1754 ms. Trace:
I20260501 13:57:34.624037 1881 raft_consensus.cc:740] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:34.623852 2108 raft_consensus.cc:515] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.624049 1899 rpcz_store.cc:276] 0501 13:57:32.868902 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.869048 (+ 146us) service_pool.cc:224] Handling call
0501 13:57:34.456076 (+1587028us) ts_tablet_manager.cc:1151] Deleting tablet 2ee580bb9d3843f9bbeae8eef2c347ad
0501 13:57:34.456117 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:34.569246 (+113129us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.615461 (+ 46215us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.623685 (+ 8224us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1586464}
I20260501 13:57:34.624698 1881 consensus_queue.cc:260] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.621430 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_320db474f1554b269265f3448e635536"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:34.625207 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 2ee580bb9d3843f9bbeae8eef2c347ad (table default.Table_abb70b9287914fbc97d3dcf78ac373bf [id=fe9f5542710e48e8bbf9a48e07c412d6]) successfully deleted
I20260501 13:57:34.625651 2108 leader_election.cc:290] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:34.626525 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_320db474f1554b269265f3448e635536 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:34.629937 1564 leader_election.cc:343] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Not found: Tablet not found: 89e2c22052d04fe1a21bcb3323385111
W20260501 13:57:34.631178 1563 leader_election.cc:343] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 89e2c22052d04fe1a21bcb3323385111
I20260501 13:57:34.631553 1563 leader_election.cc:304] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:34.633754 2108 raft_consensus.cc:2749] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:34.635480 1125 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 8d2f196f44be4fbc8059901380b71271 (table default.Table_360decd489b74f4b8ee93b008e57c45b [id=8cff6f6a2ebc447c979a45c9bfa438dc]). Expected version 2 got 0
W20260501 13:57:34.636828 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3fddeb4be4642b7b940533315be33c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:34.647553 1899 tablet_replica.cc:333] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:34.648360 1899 raft_consensus.cc:2243] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:34.648844 1899 raft_consensus.cc:2272] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:34.650583 1758 tablet_service.cc:1558] Processing DeleteTablet for tablet eeae4369be954d9d9bad846484afb6ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:34.659982 2093 ts_tablet_manager.cc:1929] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:34.664498 1899 ts_tablet_manager.cc:1916] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:34.667078 1881 ts_tablet_manager.cc:1434] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.074s user 0.006s sys 0.001s
I20260501 13:57:34.668056 1881 tablet_bootstrap.cc:492] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:34.673333 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 569a0240b1274d9bb830addbaf1ec343 (DEFAULT_TABLE table=default.Table_c9c10d0b01f846849dcd931a75ff0d63 [id=118a5cac4b274ea8879ae4b0dc467be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.674489 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569a0240b1274d9bb830addbaf1ec343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.683601 1131 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 36e15d66bf6147a08604daf2a9c53322 (table default.Table_724820186f1a45fc99ca679fdc8cc36d [id=db653ae15d0b4dada66cc4a7b2ac94bd]). Expected version 3 got 2
I20260501 13:57:34.683503 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_c3c929bb31b74edc9cf471501a43baa7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:34.687034 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3c929bb31b74edc9cf471501a43baa7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:34.719671 1127 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet e1d7a80ad4c4455a9ddee7a955e32635
I20260501 13:57:34.721740 1758 tablet_service.cc:1558] Processing DeleteTablet for tablet e1d7a80ad4c4455a9ddee7a955e32635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:34.728981 1882 tablet_bootstrap.cc:492] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:34.729045 1758 tablet_service.cc:1511] Processing CreateTablet for tablet 569a0240b1274d9bb830addbaf1ec343 (DEFAULT_TABLE table=default.Table_c9c10d0b01f846849dcd931a75ff0d63 [id=118a5cac4b274ea8879ae4b0dc467be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.730014 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569a0240b1274d9bb830addbaf1ec343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.732340 1882 ts_tablet_manager.cc:1403] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.202s user 0.009s sys 0.008s
I20260501 13:57:34.734426 2108 raft_consensus.cc:493] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:34.734891 2108 raft_consensus.cc:515] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.736653 2108 leader_election.cc:290] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:34.738746 1781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc6b0cfd20174d7680f8981bb850c50f" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:34.744359 1883 tablet_bootstrap.cc:492] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:34.744799 1883 ts_tablet_manager.cc:1403] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.160s user 0.005s sys 0.016s
I20260501 13:57:34.660375 2093 log.cc:1199] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/bf96cb4696704478a2064d6ce785d49d
I20260501 13:57:34.746487 2093 ts_tablet_manager.cc:1950] T bf96cb4696704478a2064d6ce785d49d P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:34.752755 1606 tablet_service.cc:1511] Processing CreateTablet for tablet a4265d9ab3734a8b97d745dc02c6b845 (DEFAULT_TABLE table=default.Table_4440a2fc30004b389edb91265721ba86 [id=bd84d43e8e074a9383c9843cd13e7a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.753754 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4265d9ab3734a8b97d745dc02c6b845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.758867 1883 raft_consensus.cc:359] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.759512 1883 raft_consensus.cc:385] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:34.761596 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet eaf868a5d2c24a45b183299d71f082a9
I20260501 13:57:34.763265 1899 ts_tablet_manager.cc:1929] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:34.763703 1899 log.cc:1199] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/adc47b3b2b084d33a12bd28cfad27382
I20260501 13:57:34.764286 1899 ts_tablet_manager.cc:1950] T adc47b3b2b084d33a12bd28cfad27382 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:34.765038 1607 tablet_service.cc:1558] Processing DeleteTablet for tablet e1d7a80ad4c4455a9ddee7a955e32635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:34.759781 1883 raft_consensus.cc:740] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:34.769038 1883 consensus_queue.cc:260] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.770942 1883 ts_tablet_manager.cc:1434] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.026s user 0.002s sys 0.003s
I20260501 13:57:34.716964 1460 tablet_service.cc:1511] Processing CreateTablet for tablet 569a0240b1274d9bb830addbaf1ec343 (DEFAULT_TABLE table=default.Table_c9c10d0b01f846849dcd931a75ff0d63 [id=118a5cac4b274ea8879ae4b0dc467be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.775223 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569a0240b1274d9bb830addbaf1ec343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.721465 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet e1d7a80ad4c4455a9ddee7a955e32635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:34.779014 1130 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet ec4bd4cb1e70472aa09fc93afc3b1501
I20260501 13:57:34.779731 1883 tablet_bootstrap.cc:492] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:34.780303 1130 catalog_manager.cc:6741] ReplaceTablet: tablet ec4bd4cb1e70472aa09fc93afc3b1501 deleted and replaced by tablet 8dbb619863dd405ab20ead6bbebf4d69
I20260501 13:57:34.780700 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc6b0cfd20174d7680f8981bb850c50f" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:34.782641 1564 leader_election.cc:304] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:34.788748 2152 raft_consensus.cc:2749] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:34.791086 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 102) took 2037 ms. Trace:
I20260501 13:57:34.791419 1899 rpcz_store.cc:276] 0501 13:57:32.754076 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.754201 (+ 125us) service_pool.cc:224] Handling call
0501 13:57:34.647385 (+1893184us) ts_tablet_manager.cc:1151] Deleting tablet adc47b3b2b084d33a12bd28cfad27382
0501 13:57:34.647432 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:34.717652 (+ 70220us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.763153 (+ 45501us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.791014 (+ 27861us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1732199}
I20260501 13:57:34.794198 1457 tablet_service.cc:1511] Processing CreateTablet for tablet a4265d9ab3734a8b97d745dc02c6b845 (DEFAULT_TABLE table=default.Table_4440a2fc30004b389edb91265721ba86 [id=bd84d43e8e074a9383c9843cd13e7a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.795526 1457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4265d9ab3734a8b97d745dc02c6b845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.795666 1456 tablet_service.cc:1558] Processing DeleteTablet for tablet eaf868a5d2c24a45b183299d71f082a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:34.798056 1883 tablet_bootstrap.cc:654] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:34.798648 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet ec4bd4cb1e70472aa09fc93afc3b1501 with delete_type TABLET_DATA_DELETED (replaced by tablet 8dbb619863dd405ab20ead6bbebf4d69) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:34.799144 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 56cee07054a64e199a0bfc26b63eca2a (DEFAULT_TABLE table=default.Table_2e85e5364e5f4e05b0909a6cb58843c7 [id=389132d3db474d008cc0ee9d758b4fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.799995 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56cee07054a64e199a0bfc26b63eca2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.802048 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 103) took 2038 ms. Trace:
I20260501 13:57:34.802987 1899 rpcz_store.cc:276] 0501 13:57:32.763297 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:32.763763 (+ 466us) service_pool.cc:224] Handling call
0501 13:57:34.792275 (+2028512us) ts_tablet_manager.cc:1151] Deleting tablet 9abb39c8b44d4b4c8b69751a4cdd6668
0501 13:57:34.792319 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:34.792571 (+ 252us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1876871}
I20260501 13:57:34.805233 1899 tablet_replica.cc:333] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:34.805959 1899 raft_consensus.cc:2243] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:34.806416 1899 raft_consensus.cc:2272] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:34.808140 1899 ts_tablet_manager.cc:1916] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:34.812942 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 6f6cd353ca044465a6692656412e7d82 (DEFAULT_TABLE table=default.Table_95c99739e2574437b16501060ce816af [id=a6953bc50bab4e7f92adfee0ef034465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.816581 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 19c1e0c30dce488baed9e046d90f0ba4 (DEFAULT_TABLE table=default.Table_66a4e496efa34deaa6c793a210a510ff [id=e8b19350d702499a82f41a6cce14dbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.819512 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_2e85e5364e5f4e05b0909a6cb58843c7" } new_table_name: "default.Table_9b824b59406d478884838d29a5d690f8" modify_external_catalogs: true
I20260501 13:57:34.830868 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f6cd353ca044465a6692656412e7d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.835032 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c1e0c30dce488baed9e046d90f0ba4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:34.842716 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 56cee07054a64e199a0bfc26b63eca2a consensus state has no leader
I20260501 13:57:34.842214 1129 catalog_manager.cc:5671] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "993b9ba18d8547f8af43167e727a6a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:34.843623 1129 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet c5297eb8aace46078cc3fce289fdd389 (table default.Table_eb039fccfc7d444baf4e6cdddc6be5cd [id=37089fa463db4a608b09504153e2e32a]). Expected version 1 got 0
I20260501 13:57:34.860486 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet bf96cb4696704478a2064d6ce785d49d (table default.Table_4c1ccc461d624514a37044a124d8ad6b [id=314708984ce445e2a57e799bc77d5c6f]) successfully deleted
I20260501 13:57:34.863539 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet adc47b3b2b084d33a12bd28cfad27382 (table default.Table_c4edbd4ba4c54b1e9f9909c9badd7c54 [id=1de866dfd0d145e7a13e5ae1f367524b]) successfully deleted
W20260501 13:57:34.864215 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9abb39c8b44d4b4c8b69751a4cdd6668 already in progress: creating tablet
I20260501 13:57:34.864812 1607 tablet_service.cc:1558] Processing DeleteTablet for tablet eaf868a5d2c24a45b183299d71f082a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:34.769557 1882 raft_consensus.cc:359] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.865759 1882 raft_consensus.cc:385] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:34.869117 1755 tablet_service.cc:1511] Processing CreateTablet for tablet f32170902c134c93a9aa71b2530c845e (DEFAULT_TABLE table=default.Table_6e97f9768037452593b16b51f09a18ff [id=a5184058a20f431280f6ccd10a290f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.871057 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f32170902c134c93a9aa71b2530c845e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.874018 1124 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (table default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf]). Expected version 1 got 0
I20260501 13:57:34.879544 1756 tablet_service.cc:1511] Processing CreateTablet for tablet eaf868a5d2c24a45b183299d71f082a9 (DEFAULT_TABLE table=default.Table_9f356943ea5a469c8eea733dc8231c36 [id=3780aea79eea4179b54f6cbd55f6d8f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.880761 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf868a5d2c24a45b183299d71f082a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:34.881544 1117 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 56cee07054a64e199a0bfc26b63eca2a (table default.Table_9b824b59406d478884838d29a5d690f8 [id=389132d3db474d008cc0ee9d758b4fec], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 56cee07054a64e199a0bfc26b63eca2a consensus state has no leader
I20260501 13:57:34.883301 2047 ts_tablet_manager.cc:1929] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:34.883682 2047 log.cc:1199] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/f2b59b51d81249e0ad90edba16803285
I20260501 13:57:34.883512 1754 tablet_service.cc:1511] Processing CreateTablet for tablet 967658b1be204c638fc34983293ef5ab (DEFAULT_TABLE table=default.Table_443326309dee4e12b1b00584159d6551 [id=f57f3fcda9444fa0a1ad062576f8d051]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.884299 2047 ts_tablet_manager.cc:1950] T f2b59b51d81249e0ad90edba16803285 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:34.884650 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 967658b1be204c638fc34983293ef5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.884753 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_e5f85f68afd24518b03b05a23700361d" } new_table_name: "default.Table_0d92e7b95c884bd5970e291f4189b0d9" modify_external_catalogs: true
I20260501 13:57:34.885051 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 109) took 1030 ms. Trace:
I20260501 13:57:34.885360 2047 rpcz_store.cc:276] 0501 13:57:33.855168 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:33.855317 (+ 149us) service_pool.cc:224] Handling call
0501 13:57:34.496555 (+641238us) ts_tablet_manager.cc:1151] Deleting tablet f2b59b51d81249e0ad90edba16803285
0501 13:57:34.496600 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:34.606986 (+110386us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.883179 (+276193us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:34.884997 (+ 1818us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":640636}
I20260501 13:57:34.891834 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4915b393002a47e895bfa0e9cd82da89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:34.897910 1753 tablet_service.cc:1511] Processing CreateTablet for tablet 68cb7bd1a5aa4419b9c5176a287fe885 (DEFAULT_TABLE table=default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.898149 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_dc19bcebd0934f20b6dc5a1da2f67f21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:34.899209 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cb7bd1a5aa4419b9c5176a287fe885. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:34.899578 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6650472ec4a449c48c7957a9bbec88e3 consensus state has no leader
W20260501 13:57:34.899956 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc19bcebd0934f20b6dc5a1da2f67f21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:34.904191 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4915b393002a47e895bfa0e9cd82da89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:34.914175 1757 tablet_service.cc:1511] Processing CreateTablet for tablet 19c1e0c30dce488baed9e046d90f0ba4 (DEFAULT_TABLE table=default.Table_66a4e496efa34deaa6c793a210a510ff [id=e8b19350d702499a82f41a6cce14dbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.916193 1752 tablet_service.cc:1511] Processing CreateTablet for tablet 301fad71fe6e4e53a36216c41f2850bf (DEFAULT_TABLE table=default.Table_7a4f0da973284ac49c7dd7ff3e99e09f [id=70e8e791a366491eaae840dbd7565b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.918447 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet eaf868a5d2c24a45b183299d71f082a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:34.919898 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 56cee07054a64e199a0bfc26b63eca2a (DEFAULT_TABLE table=default.Table_2e85e5364e5f4e05b0909a6cb58843c7 [id=389132d3db474d008cc0ee9d758b4fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.921486 1763 tablet_service.cc:1511] Processing CreateTablet for tablet a4265d9ab3734a8b97d745dc02c6b845 (DEFAULT_TABLE table=default.Table_4440a2fc30004b389edb91265721ba86 [id=bd84d43e8e074a9383c9843cd13e7a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.921811 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet ec4bd4cb1e70472aa09fc93afc3b1501 with delete_type TABLET_DATA_DELETED (replaced by tablet 8dbb619863dd405ab20ead6bbebf4d69) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:34.921491 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 6f6cd353ca044465a6692656412e7d82 (DEFAULT_TABLE table=default.Table_95c99739e2574437b16501060ce816af [id=a6953bc50bab4e7f92adfee0ef034465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.924443 1130 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6f9bd93befbf45c0a7415d32ce58d7fb" } modify_external_catalogs: true
I20260501 13:57:34.925113 1130 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6f9bd93befbf45c0a7415d32ce58d7fb" } modify_external_catalogs: true
I20260501 13:57:34.941083 2097 raft_consensus.cc:493] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:34.941928 2097 raft_consensus.cc:515] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.944576 2097 leader_election.cc:290] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:34.945133 2097 raft_consensus.cc:493] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:34.946481 2097 raft_consensus.cc:515] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.951264 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeae4369be954d9d9bad846484afb6ff" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:34.952384 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a0d04810f2d465ebe2964797ea30538" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:34.953279 1459 tablet_service.cc:1511] Processing CreateTablet for tablet eaf868a5d2c24a45b183299d71f082a9 (DEFAULT_TABLE table=default.Table_9f356943ea5a469c8eea733dc8231c36 [id=3780aea79eea4179b54f6cbd55f6d8f0]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:34.955295 1719 leader_election.cc:343] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:34.956156 1719 leader_election.cc:343] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:34.956795 2097 leader_election.cc:290] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:34.956894 2159 raft_consensus.cc:493] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:34.958756 2161 raft_consensus.cc:493] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:34.959208 2161 raft_consensus.cc:515] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.962594 2161 leader_election.cc:290] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:34.967569 1465 tablet_service.cc:1511] Processing CreateTablet for tablet f32170902c134c93a9aa71b2530c845e (DEFAULT_TABLE table=default.Table_6e97f9768037452593b16b51f09a18ff [id=a5184058a20f431280f6ccd10a290f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.968279 1458 tablet_service.cc:1558] Processing DeleteTablet for tablet 02926865010043e4b689c914b435400f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:34.969370 1463 tablet_service.cc:1511] Processing CreateTablet for tablet 301fad71fe6e4e53a36216c41f2850bf (DEFAULT_TABLE table=default.Table_7a4f0da973284ac49c7dd7ff3e99e09f [id=70e8e791a366491eaae840dbd7565b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.969789 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 967658b1be204c638fc34983293ef5ab (DEFAULT_TABLE table=default.Table_443326309dee4e12b1b00584159d6551 [id=f57f3fcda9444fa0a1ad062576f8d051]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.971030 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 68cb7bd1a5aa4419b9c5176a287fe885 (DEFAULT_TABLE table=default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:34.972692 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c1e0c30dce488baed9e046d90f0ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.976159 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet f2b59b51d81249e0ad90edba16803285 (table default.Table_7889b062d4464b39a2ce4ad2eef30dee [id=0444fad8c66840f2a003291fa6cc7208]) successfully deleted
I20260501 13:57:34.957355 2159 raft_consensus.cc:515] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:34.978722 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 301fad71fe6e4e53a36216c41f2850bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.979187 2159 leader_election.cc:290] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:34.979763 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4265d9ab3734a8b97d745dc02c6b845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.981869 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56cee07054a64e199a0bfc26b63eca2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.988404 1881 tablet_bootstrap.cc:654] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:34.988672 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f6cd353ca044465a6692656412e7d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.992703 2163 tablet.cc:1724] T 36e15d66bf6147a08604daf2a9c53322 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260501 13:57:35.002334 1129 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_eb039fccfc7d444baf4e6cdddc6be5cd" } modify_external_catalogs: true
I20260501 13:57:35.002898 1129 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_eb039fccfc7d444baf4e6cdddc6be5cd" } modify_external_catalogs: true
I20260501 13:57:35.007318 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4121e57a3a954fcd87c520271360371c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:35.008828 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4121e57a3a954fcd87c520271360371c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:35.009773 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 56cee07054a64e199a0bfc26b63eca2a (DEFAULT_TABLE table=default.Table_2e85e5364e5f4e05b0909a6cb58843c7 [id=389132d3db474d008cc0ee9d758b4fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.010782 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56cee07054a64e199a0bfc26b63eca2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.011900 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7127557407564f3b824f44c92babdae3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:35.013345 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7127557407564f3b824f44c92babdae3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:35.016288 1130 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 39cd02738e5f4ecfbfa06d5541614ffd
I20260501 13:57:35.021528 1458 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd02738e5f4ecfbfa06d5541614ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:35.025274 2093 tablet_replica.cc:333] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:35.026192 2093 raft_consensus.cc:2243] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:35.026775 2093 raft_consensus.cc:2272] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:35.027812 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a0d04810f2d465ebe2964797ea30538" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:35.028002 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeae4369be954d9d9bad846484afb6ff" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:35.030144 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet ec4bd4cb1e70472aa09fc93afc3b1501 with delete_type TABLET_DATA_DELETED (replaced by tablet 8dbb619863dd405ab20ead6bbebf4d69) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:35.032116 1610 tablet_service.cc:1511] Processing CreateTablet for tablet 6f6cd353ca044465a6692656412e7d82 (DEFAULT_TABLE table=default.Table_95c99739e2574437b16501060ce816af [id=a6953bc50bab4e7f92adfee0ef034465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.033185 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f6cd353ca044465a6692656412e7d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.037304 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cb7bd1a5aa4419b9c5176a287fe885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.039307 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 301fad71fe6e4e53a36216c41f2850bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.042812 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf868a5d2c24a45b183299d71f082a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.045622 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 19c1e0c30dce488baed9e046d90f0ba4 (DEFAULT_TABLE table=default.Table_66a4e496efa34deaa6c793a210a510ff [id=e8b19350d702499a82f41a6cce14dbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.046123 1130 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4120cd7643bd4ca8bdd6b6c163c8e375" } new_table_name: "default.Table_2a1ca2e987e243ad9f1c064a28866dd3" modify_external_catalogs: true
I20260501 13:57:35.046659 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c1e0c30dce488baed9e046d90f0ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.047137 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 967658b1be204c638fc34983293ef5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.047636 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f32170902c134c93a9aa71b2530c845e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:34.866034 1882 raft_consensus.cc:740] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:35.060216 1609 tablet_service.cc:1511] Processing CreateTablet for tablet eaf868a5d2c24a45b183299d71f082a9 (DEFAULT_TABLE table=default.Table_9f356943ea5a469c8eea733dc8231c36 [id=3780aea79eea4179b54f6cbd55f6d8f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.061199 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf868a5d2c24a45b183299d71f082a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.062345 1882 consensus_queue.cc:260] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.064287 1882 ts_tablet_manager.cc:1434] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.332s user 0.005s sys 0.000s
I20260501 13:57:35.070801 2108 raft_consensus.cc:493] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:35.071396 2108 raft_consensus.cc:515] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.074216 2108 leader_election.cc:290] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:35.076877 1608 tablet_service.cc:1511] Processing CreateTablet for tablet f32170902c134c93a9aa71b2530c845e (DEFAULT_TABLE table=default.Table_6e97f9768037452593b16b51f09a18ff [id=a5184058a20f431280f6ccd10a290f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.077533 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 967658b1be204c638fc34983293ef5ab (DEFAULT_TABLE table=default.Table_443326309dee4e12b1b00584159d6551 [id=f57f3fcda9444fa0a1ad062576f8d051]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.078176 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f32170902c134c93a9aa71b2530c845e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.078827 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 967658b1be204c638fc34983293ef5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.081231 1129 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet c5297eb8aace46078cc3fce289fdd389
W20260501 13:57:35.100026 1119 consensus_peers.cc:597] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b -> Peer 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815): Couldn't send request to peer 50c0932256e74399a537670bb424a1b7. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260501 13:57:35.102670 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_609f259197a542caaa0dd2fdc258b986" } new_table_name: "default.Table_759f3009542147de94c73784e1698246" modify_external_catalogs: true
I20260501 13:57:35.103015 1602 tablet_service.cc:1511] Processing CreateTablet for tablet 301fad71fe6e4e53a36216c41f2850bf (DEFAULT_TABLE table=default.Table_7a4f0da973284ac49c7dd7ff3e99e09f [id=70e8e791a366491eaae840dbd7565b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.106060 1882 tablet_bootstrap.cc:492] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:35.114251 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 68cb7bd1a5aa4419b9c5176a287fe885 (DEFAULT_TABLE table=default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.122193 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_65c41fbebfea4a3c8bacf216d517e869"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:35.127856 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 301fad71fe6e4e53a36216c41f2850bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.108292 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 115) took 1028 ms. Trace:
I20260501 13:57:35.130864 2047 rpcz_store.cc:276] 0501 13:57:34.079362 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:34.079462 (+ 100us) service_pool.cc:224] Handling call
0501 13:57:34.886421 (+806959us) ts_tablet_manager.cc:1151] Deleting tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb
0501 13:57:34.919438 (+ 33017us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:35.107253 (+187815us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":799769}
W20260501 13:57:35.154946 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb already in progress: creating tablet
I20260501 13:57:35.166810 1882 tablet_bootstrap.cc:654] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:35.132658 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 116) took 1022 ms. Trace:
I20260501 13:57:35.175179 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d2f196f44be4fbc8059901380b71271" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:35.178310 1421 leader_election.cc:343] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:35.180387 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c2d5336bfec43dbb38ce7071834bca3" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:35.184240 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd02738e5f4ecfbfa06d5541614ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:35.173086 2047 rpcz_store.cc:276] 0501 13:57:34.110171 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:34.110303 (+ 132us) service_pool.cc:224] Handling call
0501 13:57:35.132246 (+1021943us) ts_tablet_manager.cc:1151] Deleting tablet 863a10cfd14c4694b7c6141c0851167a
0501 13:57:35.132295 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:35.132595 (+ 300us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1021391}
I20260501 13:57:35.195590 2047 tablet_replica.cc:333] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:35.196493 2047 raft_consensus.cc:2243] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:35.196988 2047 raft_consensus.cc:2272] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:35.209426 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 863a10cfd14c4694b7c6141c0851167a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 863a10cfd14c4694b7c6141c0851167a already in progress: creating tablet
W20260501 13:57:35.210074 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet a9672aab85c64d548a4f6583a1e9af20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9672aab85c64d548a4f6583a1e9af20 already in progress: creating tablet
I20260501 13:57:35.208844 1882 tablet_bootstrap.cc:492] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:35.213732 1882 ts_tablet_manager.cc:1403] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.109s user 0.003s sys 0.014s
W20260501 13:57:35.216512 1421 leader_election.cc:343] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:35.216590 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_907a34a78d554ca8a54102bcf65f7ed7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:35.220731 1125 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_041fce4b819d4a4c80c58137b4213234" } modify_external_catalogs: true
I20260501 13:57:35.221238 1125 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_041fce4b819d4a4c80c58137b4213234" } modify_external_catalogs: true
W20260501 13:57:35.224750 1564 leader_election.cc:343] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Not found: Tablet not found: f2b59b51d81249e0ad90edba16803285
W20260501 13:57:35.238701 1717 leader_election.cc:343] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:35.246017 1717 leader_election.cc:304] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:35.246654 1123 master_service.cc:954] ReplaceTablet: received request to replace tablet b507a4d4c9c54bc68d92d9286794a264 from {username='slave'} at 127.0.0.1:41700
W20260501 13:57:35.247061 1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 64843.9us
W20260501 13:57:35.247856 1717 leader_election.cc:343] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:35.248253 1717 leader_election.cc:304] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:35.252436 1882 raft_consensus.cc:359] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.253350 1882 raft_consensus.cc:385] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:35.254020 1882 raft_consensus.cc:740] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:35.254696 1882 consensus_queue.cc:260] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.255762 2093 ts_tablet_manager.cc:1916] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:35.260833 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet e1d7a80ad4c4455a9ddee7a955e32635 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1d7a80ad4c4455a9ddee7a955e32635 already in progress: creating tablet
W20260501 13:57:35.261508 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet eaf868a5d2c24a45b183299d71f082a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eaf868a5d2c24a45b183299d71f082a9 already in progress: creating tablet
W20260501 13:57:35.262181 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet ec4bd4cb1e70472aa09fc93afc3b1501 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec4bd4cb1e70472aa09fc93afc3b1501 already in progress: creating tablet
I20260501 13:57:35.266217 2144 raft_consensus.cc:2749] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:35.264549 2097 raft_consensus.cc:2749] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:35.276516 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd02738e5f4ecfbfa06d5541614ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:35.281420 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d2f196f44be4fbc8059901380b71271" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:35.282212 1881 tablet_bootstrap.cc:492] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:35.282639 1881 ts_tablet_manager.cc:1403] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.615s user 0.008s sys 0.006s
I20260501 13:57:35.282954 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cb7bd1a5aa4419b9c5176a287fe885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.284868 1881 raft_consensus.cc:359] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.285459 1881 raft_consensus.cc:385] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:35.285712 1881 raft_consensus.cc:740] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:35.282102 1780 raft_consensus.cc:2468] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
W20260501 13:57:35.296589 1563 leader_election.cc:343] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: f2b59b51d81249e0ad90edba16803285
I20260501 13:57:35.297096 1563 leader_election.cc:304] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:35.168521 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet c5297eb8aace46078cc3fce289fdd389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:35.303782 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65c41fbebfea4a3c8bacf216d517e869 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:35.305064 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_907a34a78d554ca8a54102bcf65f7ed7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:35.310245 2163 tablet.cc:1724] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260501 13:57:35.316339 2164 tablet.cc:1724] T 36e15d66bf6147a08604daf2a9c53322 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260501 13:57:35.327633 1418 leader_election.cc:304] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:35.328363 1931 raft_consensus.cc:2804] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:35.328735 1931 raft_consensus.cc:493] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:35.329052 1931 raft_consensus.cc:3060] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:35.329351 2174 tablet.cc:1724] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260501 13:57:35.330391 1882 ts_tablet_manager.cc:1434] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.116s user 0.002s sys 0.005s
I20260501 13:57:35.331447 1882 tablet_bootstrap.cc:492] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:35.336094 1931 raft_consensus.cc:515] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:35.338819 1130 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 consensus state has no leader
I20260501 13:57:35.344981 1779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c2d5336bfec43dbb38ce7071834bca3" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:35.346100 1779 raft_consensus.cc:2468] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:35.343039 2047 ts_tablet_manager.cc:1916] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:35.340023 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d2f196f44be4fbc8059901380b71271" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
W20260501 13:57:35.343878 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 47350.7us
I20260501 13:57:35.344327 2152 raft_consensus.cc:2749] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:35.354406 1608 tablet_service.cc:1558] Processing DeleteTablet for tablet f2b59b51d81249e0ad90edba16803285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:28 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:35.327652 2171 tablet.cc:1724] T 36e15d66bf6147a08604daf2a9c53322 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20260501 13:57:35.369848 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 406e1af8b7d9488f88697206daa87c05 consensus state has no leader
I20260501 13:57:35.370764 1931 leader_election.cc:290] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:35.377933 1629 raft_consensus.cc:3060] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:35.372133 1881 consensus_queue.cc:260] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.383980 2166 raft_consensus.cc:493] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:35.384433 2166 raft_consensus.cc:515] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.388365 2166 leader_election.cc:290] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:35.394695 1883 tablet_bootstrap.cc:492] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:35.395179 1883 ts_tablet_manager.cc:1403] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.616s user 0.009s sys 0.007s
I20260501 13:57:35.397063 1418 leader_election.cc:304] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:35.416561 1881 ts_tablet_manager.cc:1434] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.134s user 0.005s sys 0.001s
I20260501 13:57:35.417397 2162 raft_consensus.cc:2804] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:35.417737 2162 raft_consensus.cc:493] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:35.418022 2162 raft_consensus.cc:3060] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:35.361186 1899 ts_tablet_manager.cc:1929] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:35.421285 1778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d2f196f44be4fbc8059901380b71271" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:35.423252 1778 raft_consensus.cc:3060] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:35.424589 1899 log.cc:1199] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/2ff05404061447199d936d005afda653
I20260501 13:57:35.427580 1779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7645593dc184dab9f677f9371a88354" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:35.428162 1779 raft_consensus.cc:2468] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:35.433602 1778 raft_consensus.cc:2468] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:35.436636 1883 raft_consensus.cc:359] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.437256 1883 raft_consensus.cc:385] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:35.437533 1883 raft_consensus.cc:740] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:35.438090 1883 consensus_queue.cc:260] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.460115 2097 raft_consensus.cc:493] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:35.460553 2097 raft_consensus.cc:515] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.463284 1418 leader_election.cc:304] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:35.464254 1881 tablet_bootstrap.cc:492] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:35.464118 1418 leader_election.cc:304] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:35.464421 2166 raft_consensus.cc:2804] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:35.464744 2166 raft_consensus.cc:493] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:35.462265 2097 leader_election.cc:290] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:35.466768 1931 raft_consensus.cc:2804] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:35.467303 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9c92846d4e4255a1c8ce3835a0a18d" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:35.468792 2166 raft_consensus.cc:3060] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:35.473055 1899 ts_tablet_manager.cc:1950] T 2ff05404061447199d936d005afda653 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:35.475258 1883 ts_tablet_manager.cc:1434] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.080s user 0.006s sys 0.000s
I20260501 13:57:35.476583 1883 tablet_bootstrap.cc:492] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:35.477106 2166 raft_consensus.cc:515] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.482241 1132 master_service.cc:954] ReplaceTablet: received request to replace tablet f32170902c134c93a9aa71b2530c845e from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:35.486810 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 113) took 2103 ms. Trace:
I20260501 13:57:35.487140 1899 rpcz_store.cc:276] 0501 13:57:33.383173 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:33.383630 (+ 457us) service_pool.cc:224] Handling call
0501 13:57:34.805075 (+1421445us) ts_tablet_manager.cc:1151] Deleting tablet 2ff05404061447199d936d005afda653
0501 13:57:34.805118 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:35.324972 (+519854us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:35.360973 (+ 36001us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:35.486739 (+125766us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1420850}
I20260501 13:57:35.471105 1931 raft_consensus.cc:697] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
W20260501 13:57:35.490010 1717 leader_election.cc:343] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:35.490872 2159 consensus_queue.cc:1048] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:57:35.488878 1931 consensus_queue.cc:237] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.385345 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet c5297eb8aace46078cc3fce289fdd389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:35.498658 2047 ts_tablet_manager.cc:1929] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:35.499007 2047 log.cc:1199] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/02926865010043e4b689c914b435400f
I20260501 13:57:35.499567 2047 ts_tablet_manager.cc:1950] T 02926865010043e4b689c914b435400f P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:35.501489 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 02926865010043e4b689c914b435400f (table default.Table_4bb39ebdce2145ccaf94f17bb84197b8 [id=a7c0d6e58eac4f9aabd0f2bd585db9a6]) successfully deleted
I20260501 13:57:35.501744 2047 tablet_replica.cc:333] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:35.502251 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7645593dc184dab9f677f9371a88354" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:35.507885 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_db50865558a14081b6f0280d9f45ceea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:35.508985 2047 raft_consensus.cc:2243] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
W20260501 13:57:35.509383 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db50865558a14081b6f0280d9f45ceea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:35.509586 2047 pending_rounds.cc:70] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: Trying to abort 1 pending ops.
I20260501 13:57:35.509794 2047 pending_rounds.cc:77] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229228496216064 op_type: NO_OP noop_request { }
I20260501 13:57:35.510111 2047 raft_consensus.cc:2889] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:35.510399 2047 raft_consensus.cc:2272] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:35.511663 1899 tablet_replica.cc:333] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:35.512377 1899 raft_consensus.cc:2243] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:35.512941 1899 raft_consensus.cc:2272] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:35.512933 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 39cd02738e5f4ecfbfa06d5541614ffd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39cd02738e5f4ecfbfa06d5541614ffd already in progress: creating tablet
I20260501 13:57:35.514300 2047 ts_tablet_manager.cc:1916] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:35.514309 1124 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_427f541417f845efb488354adcc40139" } modify_external_catalogs: true
I20260501 13:57:35.514819 1124 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_427f541417f845efb488354adcc40139" } modify_external_catalogs: true
I20260501 13:57:35.515403 1899 ts_tablet_manager.cc:1916] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:35.522153 1629 raft_consensus.cc:2468] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:35.529897 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 2ff05404061447199d936d005afda653 (table default.Table_26541c8dd6a347a78e26e6cb55fa3b06 [id=15dd9506496343159cc3f4de3afcef5e]) successfully deleted
I20260501 13:57:35.531409 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet c5297eb8aace46078cc3fce289fdd389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:35.535663 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9c92846d4e4255a1c8ce3835a0a18d" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:35.539364 1719 leader_election.cc:343] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:35.544027 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_0abda464a00b44f49271dbde4c90e526"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:35.544957 1719 leader_election.cc:304] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:35.545547 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0abda464a00b44f49271dbde4c90e526 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:35.545804 1883 tablet_bootstrap.cc:654] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:35.546862 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_5e1da2dc1631431c9ced9ebb18d903d8" } new_table_name: "default.Table_0c2d991875174eefbc2fa69108cc7082" modify_external_catalogs: true
I20260501 13:57:35.555061 2172 raft_consensus.cc:2749] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:35.564049 1123 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (table default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf]). Expected version 1 got 0
W20260501 13:57:35.564091 1421 leader_election.cc:343] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:35.564303 2047 ts_tablet_manager.cc:1929] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:35.564597 2047 log.cc:1199] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/c5297eb8aace46078cc3fce289fdd389
I20260501 13:57:35.565240 2047 ts_tablet_manager.cc:1950] T c5297eb8aace46078cc3fce289fdd389 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:35.567165 1130 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 8d2f196f44be4fbc8059901380b71271 (table default.Table_360decd489b74f4b8ee93b008e57c45b [id=8cff6f6a2ebc447c979a45c9bfa438dc]). Expected version 2 got 0
I20260501 13:57:35.567420 1883 tablet_bootstrap.cc:492] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:35.567791 1883 ts_tablet_manager.cc:1403] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.000s
I20260501 13:57:35.566212 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet c5297eb8aace46078cc3fce289fdd389 (table default.Table_eb039fccfc7d444baf4e6cdddc6be5cd [id=37089fa463db4a608b09504153e2e32a]) successfully deleted
I20260501 13:57:35.577557 1881 tablet_bootstrap.cc:654] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:35.578879 1606 tablet_service.cc:1511] Processing CreateTablet for tablet f52ea8fb8ae0427d804001aed42cd67c (DEFAULT_TABLE table=default.Table_fb6bb57c871e4855aba95f3b430a3fc7 [id=09adc59479f44d358ed186c100b690b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.579952 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52ea8fb8ae0427d804001aed42cd67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.583978 1123 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c3fddeb4be4642b7b940533315be33c1" } new_table_name: "default.Table_6fe081b50a0640afbbb524ff85b821d2" modify_external_catalogs: true
I20260501 13:57:35.587085 2162 raft_consensus.cc:515] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.587180 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_200f07afb8cf4eafbace88f5cd3090d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:35.588604 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_200f07afb8cf4eafbace88f5cd3090d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:35.591917 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c2d5336bfec43dbb38ce7071834bca3" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:35.590215 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7645593dc184dab9f677f9371a88354" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:35.593747 1779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c2d5336bfec43dbb38ce7071834bca3" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:35.594300 1779 raft_consensus.cc:3060] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:35.595167 1421 leader_election.cc:343] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:35.595779 1421 leader_election.cc:343] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:35.595811 2162 leader_election.cc:290] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:35.601778 2164 tablet.cc:1724] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260501 13:57:35.605121 1778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7645593dc184dab9f677f9371a88354" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:35.605775 1778 raft_consensus.cc:3060] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:35.607599 2166 leader_election.cc:290] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:35.554603 2093 ts_tablet_manager.cc:1929] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:35.612169 2093 log.cc:1199] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/89e2c22052d04fe1a21bcb3323385111
I20260501 13:57:35.612849 2093 ts_tablet_manager.cc:1950] T 89e2c22052d04fe1a21bcb3323385111 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:35.618836 1779 raft_consensus.cc:2468] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:35.620136 1418 leader_election.cc:304] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:35.621706 2166 raft_consensus.cc:2804] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:35.622301 2166 raft_consensus.cc:697] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:35.623214 2166 consensus_queue.cc:237] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.623730 1629 raft_consensus.cc:3060] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:35.636832 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 126) took 1287 ms. Trace:
I20260501 13:57:35.637207 2093 rpcz_store.cc:276] 0501 13:57:34.349350 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:34.349456 (+ 106us) service_pool.cc:224] Handling call
0501 13:57:35.025099 (+675643us) ts_tablet_manager.cc:1151] Deleting tablet 89e2c22052d04fe1a21bcb3323385111
0501 13:57:35.025146 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:35.532474 (+507328us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:35.554444 (+ 21970us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:35.636754 (+ 82310us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":675092}
I20260501 13:57:35.638435 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 89e2c22052d04fe1a21bcb3323385111 (table default.Table_4b9a916d6a754ac7b38819b39d0b1671 [id=36e8cb1c80cc45e7b4c501608c374af9]) successfully deleted
I20260501 13:57:35.646539 1883 raft_consensus.cc:359] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.647171 1883 raft_consensus.cc:385] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:35.647406 1883 raft_consensus.cc:740] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:35.648162 1883 consensus_queue.cc:260] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.649230 1456 tablet_service.cc:1511] Processing CreateTablet for tablet f52ea8fb8ae0427d804001aed42cd67c (DEFAULT_TABLE table=default.Table_fb6bb57c871e4855aba95f3b430a3fc7 [id=09adc59479f44d358ed186c100b690b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.650636 1752 tablet_service.cc:1511] Processing CreateTablet for tablet f52ea8fb8ae0427d804001aed42cd67c (DEFAULT_TABLE table=default.Table_fb6bb57c871e4855aba95f3b430a3fc7 [id=09adc59479f44d358ed186c100b690b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.651659 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52ea8fb8ae0427d804001aed42cd67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.651412 1463 tablet_service.cc:1511] Processing CreateTablet for tablet 8dbb619863dd405ab20ead6bbebf4d69 (DEFAULT_TABLE table=default.Table_6ee51a9fc8ae4340b14c506cf9c450c2 [id=e61fe8a70f66497db17db0afacc35100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.653561 1882 tablet_bootstrap.cc:654] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:35.662916 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet d5d4064154c744198fa438cd3e3b7288
I20260501 13:57:35.666177 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet d5d4064154c744198fa438cd3e3b7288 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:35.666929 2047 tablet_replica.cc:333] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:35.667577 2047 raft_consensus.cc:2243] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:35.667773 1755 tablet_service.cc:1558] Processing DeleteTablet for tablet d5d4064154c744198fa438cd3e3b7288 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:35.668145 2047 raft_consensus.cc:2272] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:35.669850 2047 ts_tablet_manager.cc:1916] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:35.672346 1778 raft_consensus.cc:2468] T f7645593dc184dab9f677f9371a88354 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:35.677377 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet d5d4064154c744198fa438cd3e3b7288 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:35.678207 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet e1d7a80ad4c4455a9ddee7a955e32635 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1d7a80ad4c4455a9ddee7a955e32635 already in progress: creating tablet
W20260501 13:57:35.681507 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet eaf868a5d2c24a45b183299d71f082a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eaf868a5d2c24a45b183299d71f082a9 already in progress: creating tablet
W20260501 13:57:35.682484 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet ec4bd4cb1e70472aa09fc93afc3b1501 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec4bd4cb1e70472aa09fc93afc3b1501 already in progress: creating tablet
W20260501 13:57:35.683199 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 39cd02738e5f4ecfbfa06d5541614ffd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39cd02738e5f4ecfbfa06d5541614ffd already in progress: creating tablet
I20260501 13:57:35.683466 2093 tablet_replica.cc:333] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:35.684237 2093 raft_consensus.cc:2243] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:35.684772 2093 raft_consensus.cc:2272] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:35.687270 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_70769641e9fe4b9f9af063e534d0516c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:35.688782 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70769641e9fe4b9f9af063e534d0516c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:35.711166 1418 leader_election.cc:304] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:35.711954 2166 raft_consensus.cc:2804] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:35.712380 2166 raft_consensus.cc:697] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:35.713083 2166 consensus_queue.cc:237] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.720280 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 8dbb619863dd405ab20ead6bbebf4d69 (DEFAULT_TABLE table=default.Table_6ee51a9fc8ae4340b14c506cf9c450c2 [id=e61fe8a70f66497db17db0afacc35100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.721264 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dbb619863dd405ab20ead6bbebf4d69. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:35.722971 1128 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69626ed095bd4df7859563c176dafffc consensus state has no leader
I20260501 13:57:35.730512 1883 ts_tablet_manager.cc:1434] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.162s user 0.008s sys 0.000s
I20260501 13:57:35.732106 1883 tablet_bootstrap.cc:492] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:35.741809 1899 ts_tablet_manager.cc:1929] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260501 13:57:35.742177 1899 log.cc:1199] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/921e5c3eaaa44c62abbc90f560049614
I20260501 13:57:35.743639 1899 ts_tablet_manager.cc:1950] T 921e5c3eaaa44c62abbc90f560049614 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:35.745410 2093 ts_tablet_manager.cc:1916] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:35.748329 1882 tablet_bootstrap.cc:492] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:35.748831 1882 ts_tablet_manager.cc:1403] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.418s user 0.005s sys 0.007s
I20260501 13:57:35.748850 1881 tablet_bootstrap.cc:492] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:35.749238 1881 ts_tablet_manager.cc:1403] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.285s user 0.008s sys 0.011s
I20260501 13:57:35.752112 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52ea8fb8ae0427d804001aed42cd67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.754814 1602 tablet_service.cc:1511] Processing CreateTablet for tablet 4f7db275918c41139cf9bf44bba001a7 (DEFAULT_TABLE table=default.Table_eeb1c65f0d0448cba002f35e5a5aeb5d [id=48f8fa7db51c4934b64dda625167ac63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.755592 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 4f7db275918c41139cf9bf44bba001a7 (DEFAULT_TABLE table=default.Table_eeb1c65f0d0448cba002f35e5a5aeb5d [id=48f8fa7db51c4934b64dda625167ac63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.756757 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f7db275918c41139cf9bf44bba001a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.762250 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f7db275918c41139cf9bf44bba001a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.766772 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dbb619863dd405ab20ead6bbebf4d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.773578 1881 raft_consensus.cc:359] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.774195 1881 raft_consensus.cc:385] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:35.774438 1881 raft_consensus.cc:740] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:35.789860 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 2c8d5d04a19443cfaa47b733b1d6ada6 (DEFAULT_TABLE table=default.Table_cc6986b178ed47349a3dfda0ef76ce70 [id=b0864c2d4d764c42ba3ec7a2bffc13b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.790202 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 2c8d5d04a19443cfaa47b733b1d6ada6 (DEFAULT_TABLE table=default.Table_cc6986b178ed47349a3dfda0ef76ce70 [id=b0864c2d4d764c42ba3ec7a2bffc13b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.790921 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c8d5d04a19443cfaa47b733b1d6ada6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.791136 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c8d5d04a19443cfaa47b733b1d6ada6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.798851 1881 consensus_queue.cc:260] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.813009 1125 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90
I20260501 13:57:35.816313 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:35.816398 1882 raft_consensus.cc:359] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.817019 1882 raft_consensus.cc:385] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:35.817279 1882 raft_consensus.cc:740] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:35.784094 1755 tablet_service.cc:1511] Processing CreateTablet for tablet 4f7db275918c41139cf9bf44bba001a7 (DEFAULT_TABLE table=default.Table_eeb1c65f0d0448cba002f35e5a5aeb5d [id=48f8fa7db51c4934b64dda625167ac63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.821018 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f7db275918c41139cf9bf44bba001a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.823405 1881 ts_tablet_manager.cc:1434] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.074s user 0.007s sys 0.000s
I20260501 13:57:35.806299 1883 tablet_bootstrap.cc:654] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:35.830745 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:35.831188 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 0468f84c6d1b4440b82faebd8a33fc9f (DEFAULT_TABLE table=default.Table_d17e515649da406b814a3e991d8f0d41 [id=8d69802ac4b648c185a3cc65acfa581b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.832445 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0468f84c6d1b4440b82faebd8a33fc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.834864 1881 tablet_bootstrap.cc:492] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:35.787233 1752 tablet_service.cc:1511] Processing CreateTablet for tablet 8dbb619863dd405ab20ead6bbebf4d69 (DEFAULT_TABLE table=default.Table_6ee51a9fc8ae4340b14c506cf9c450c2 [id=e61fe8a70f66497db17db0afacc35100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.802842 1754 tablet_service.cc:1511] Processing CreateTablet for tablet 2c8d5d04a19443cfaa47b733b1d6ada6 (DEFAULT_TABLE table=default.Table_cc6986b178ed47349a3dfda0ef76ce70 [id=b0864c2d4d764c42ba3ec7a2bffc13b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.836154 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c8d5d04a19443cfaa47b733b1d6ada6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.811981 2097 raft_consensus.cc:493] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:35.838665 2097 raft_consensus.cc:515] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.815501 1757 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:35.837088 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 0468f84c6d1b4440b82faebd8a33fc9f (DEFAULT_TABLE table=default.Table_d17e515649da406b814a3e991d8f0d41 [id=8d69802ac4b648c185a3cc65acfa581b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.842765 2097 leader_election.cc:290] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:35.843832 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "863a10cfd14c4694b7c6141c0851167a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:35.845193 1717 leader_election.cc:343] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:35.846127 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0468f84c6d1b4440b82faebd8a33fc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.851630 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "863a10cfd14c4694b7c6141c0851167a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:35.853063 1719 leader_election.cc:343] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:35.853427 1719 leader_election.cc:304] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:35.854013 2097 raft_consensus.cc:2749] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:35.855839 1757 tablet_service.cc:1511] Processing CreateTablet for tablet 0468f84c6d1b4440b82faebd8a33fc9f (DEFAULT_TABLE table=default.Table_d17e515649da406b814a3e991d8f0d41 [id=8d69802ac4b648c185a3cc65acfa581b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.856860 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0468f84c6d1b4440b82faebd8a33fc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.864534 1125 master_service.cc:954] ReplaceTablet: received request to replace tablet 69626ed095bd4df7859563c176dafffc from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:35.865207 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dbb619863dd405ab20ead6bbebf4d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.869240 1881 tablet_bootstrap.cc:654] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:35.869612 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 1ab056da506f454984a07026341c534b (DEFAULT_TABLE table=default.Table_a27ca155bb494a99bc23aefa4e6823f2 [id=ec264b057ec84337b6e8e237bb31b98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.870569 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ab056da506f454984a07026341c534b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.872668 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 1ab056da506f454984a07026341c534b (DEFAULT_TABLE table=default.Table_a27ca155bb494a99bc23aefa4e6823f2 [id=ec264b057ec84337b6e8e237bb31b98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.873687 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ab056da506f454984a07026341c534b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.875005 1129 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_cc6986b178ed47349a3dfda0ef76ce70" } modify_external_catalogs: true
I20260501 13:57:35.875505 1129 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_cc6986b178ed47349a3dfda0ef76ce70" } modify_external_catalogs: true
I20260501 13:57:35.877051 1602 tablet_service.cc:1511] Processing CreateTablet for tablet 50ec3d3f3eb449759a8df52279b2e5e8 (DEFAULT_TABLE table=default.Table_c3fddeb4be4642b7b940533315be33c1 [id=32fd27a2bdec45698663dac24dd99b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.878093 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ec3d3f3eb449759a8df52279b2e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.879468 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 1ab056da506f454984a07026341c534b (DEFAULT_TABLE table=default.Table_a27ca155bb494a99bc23aefa4e6823f2 [id=ec264b057ec84337b6e8e237bb31b98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.880508 1883 tablet_bootstrap.cc:492] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:35.880597 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ab056da506f454984a07026341c534b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.880927 1883 ts_tablet_manager.cc:1403] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.149s user 0.009s sys 0.000s
I20260501 13:57:35.880686 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_430d0393c555431e8d75edfd188a80d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:35.882087 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_430d0393c555431e8d75edfd188a80d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:35.886514 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 112) took 2504 ms. Trace:
I20260501 13:57:35.886924 1899 rpcz_store.cc:276] 0501 13:57:33.381871 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:33.382016 (+ 145us) service_pool.cc:224] Handling call
0501 13:57:35.511482 (+2129466us) ts_tablet_manager.cc:1151] Deleting tablet 921e5c3eaaa44c62abbc90f560049614
0501 13:57:35.511547 (+ 65us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:35.634908 (+123361us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:35.741686 (+106778us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:35.886417 (+144731us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2066889}
I20260501 13:57:35.887373 2047 ts_tablet_manager.cc:1929] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260501 13:57:35.887732 2047 log.cc:1199] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/d5d4064154c744198fa438cd3e3b7288
I20260501 13:57:35.888181 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 921e5c3eaaa44c62abbc90f560049614 (table default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47]) successfully deleted
I20260501 13:57:35.888373 2047 ts_tablet_manager.cc:1950] T d5d4064154c744198fa438cd3e3b7288 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:35.890498 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 50ec3d3f3eb449759a8df52279b2e5e8 (DEFAULT_TABLE table=default.Table_c3fddeb4be4642b7b940533315be33c1 [id=32fd27a2bdec45698663dac24dd99b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.891399 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ec3d3f3eb449759a8df52279b2e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.912444 1608 tablet_service.cc:1511] Processing CreateTablet for tablet 31c73582788648aab55d678ad6e4fc76 (DEFAULT_TABLE table=default.Table_c3c929bb31b74edc9cf471501a43baa7 [id=ee3db9bd7d494e5b836198707548c957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.913542 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c73582788648aab55d678ad6e4fc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.913326 1882 consensus_queue.cc:260] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.913908 1754 tablet_service.cc:1511] Processing CreateTablet for tablet 50ec3d3f3eb449759a8df52279b2e5e8 (DEFAULT_TABLE table=default.Table_c3fddeb4be4642b7b940533315be33c1 [id=32fd27a2bdec45698663dac24dd99b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:35.916137 1899 tablet_replica.cc:333] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:35.916889 1899 raft_consensus.cc:2243] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:35.917352 1899 raft_consensus.cc:2272] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:35.918587 1882 ts_tablet_manager.cc:1434] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.169s user 0.006s sys 0.000s
I20260501 13:57:35.919517 1882 tablet_bootstrap.cc:492] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:35.923297 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ec3d3f3eb449759a8df52279b2e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:35.922545 1883 raft_consensus.cc:359] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.928882 1883 raft_consensus.cc:385] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:35.929172 1883 raft_consensus.cc:740] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:35.932366 1899 ts_tablet_manager.cc:1916] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:35.935498 1118 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 69626ed095bd4df7859563c176dafffc (table default.Table_0c2d991875174eefbc2fa69108cc7082 [id=63f1fe234b1143869f21b76118b58234], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 69626ed095bd4df7859563c176dafffc consensus state has no leader
I20260501 13:57:35.942052 1127 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_d56638cc43fa42f1a0da405e69383a6a [id=b8901f06222746b8bf418391866b92cc] alter complete (version 3)
I20260501 13:57:35.943816 2093 ts_tablet_manager.cc:1929] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:35.943825 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_299166301fa440eca9a509c07bde6ab3" } new_table_name: "default.Table_7591dec5d3bf43b8b63db3443295ed26" modify_external_catalogs: true
I20260501 13:57:35.944191 2093 log.cc:1199] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/f2b59b51d81249e0ad90edba16803285
I20260501 13:57:35.944849 2093 ts_tablet_manager.cc:1950] T f2b59b51d81249e0ad90edba16803285 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:35.943342 1883 consensus_queue.cc:260] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.952665 1883 ts_tablet_manager.cc:1434] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.071s user 0.006s sys 0.000s
I20260501 13:57:35.954120 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_427f541417f845efb488354adcc40139 [id=0dc2b91768f84b4d93b18f304d31883e]) successfully deleted
I20260501 13:57:35.966075 1132 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet f32170902c134c93a9aa71b2530c845e
I20260501 13:57:35.967569 1132 catalog_manager.cc:6741] ReplaceTablet: tablet f32170902c134c93a9aa71b2530c845e deleted and replaced by tablet 3b0ed592fd3642c194bb946c8c6003b4
I20260501 13:57:35.971355 1130 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_724820186f1a45fc99ca679fdc8cc36d [id=db653ae15d0b4dada66cc4a7b2ac94bd] alter complete (version 3)
I20260501 13:57:35.973430 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet f32170902c134c93a9aa71b2530c845e with delete_type TABLET_DATA_DELETED (replaced by tablet 3b0ed592fd3642c194bb946c8c6003b4) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:35.974550 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet f32170902c134c93a9aa71b2530c845e with delete_type TABLET_DATA_DELETED (replaced by tablet 3b0ed592fd3642c194bb946c8c6003b4) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:35.977497 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet f32170902c134c93a9aa71b2530c845e with delete_type TABLET_DATA_DELETED (replaced by tablet 3b0ed592fd3642c194bb946c8c6003b4) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:35.981585 1883 tablet_bootstrap.cc:492] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:35.982846 2166 raft_consensus.cc:493] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:35.983295 2166 raft_consensus.cc:515] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:35.985899 2166 leader_election.cc:290] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:35.986268 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e86d6c935d6444b8a953e7be1bdfa51" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:35.987967 2097 raft_consensus.cc:493] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:35.988171 1778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e86d6c935d6444b8a953e7be1bdfa51" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:35.988425 2097 raft_consensus.cc:515] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:35.989672 1421 leader_election.cc:343] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:35.993878 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e86d6c935d6444b8a953e7be1bdfa51" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:35.990228 2097 leader_election.cc:290] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:35.995927 1778 raft_consensus.cc:2468] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:35.999416 1418 leader_election.cc:304] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:36.000648 2166 raft_consensus.cc:2804] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:36.000988 2166 raft_consensus.cc:493] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:36.001297 2166 raft_consensus.cc:3060] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:36.004920 1719 leader_election.cc:343] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.015619 1882 tablet_bootstrap.cc:654] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:36.028407 2166 raft_consensus.cc:515] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.029139 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet f2b59b51d81249e0ad90edba16803285 (table default.Table_7889b062d4464b39a2ce4ad2eef30dee [id=0444fad8c66840f2a003291fa6cc7208]) successfully deleted
I20260501 13:57:36.028905 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_22d1c2cee41e4e9dad9e52bde00f1524"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:36.034160 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 6c2d5336bfec43dbb38ce7071834bca3 (table default.Table_66545ca4c3d540ada57273201e17f2d8 [id=697eaa78d5e541988cd48466c64c5915]). Expected version 1 got 0
I20260501 13:57:36.035763 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e86d6c935d6444b8a953e7be1bdfa51" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
W20260501 13:57:36.037418 1421 leader_election.cc:343] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:36.038625 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22d1c2cee41e4e9dad9e52bde00f1524 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:36.039147 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_945a3cdaaa824caab261338095683b43" } modify_external_catalogs: true
I20260501 13:57:36.039703 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_945a3cdaaa824caab261338095683b43" } modify_external_catalogs: true
I20260501 13:57:36.043040 1130 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 6650472ec4a449c48c7957a9bbec88e3 (table default.Table_0d92e7b95c884bd5970e291f4189b0d9 [id=774a7eb90ba449b6aef225d819678439]). Expected version 1 got 0
W20260501 13:57:36.055567 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet c5297eb8aace46078cc3fce289fdd389 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5297eb8aace46078cc3fce289fdd389 already in progress: creating tablet
I20260501 13:57:36.058573 1881 tablet_bootstrap.cc:492] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:36.058573 1882 tablet_bootstrap.cc:492] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:36.059000 1881 ts_tablet_manager.cc:1403] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.235s user 0.012s sys 0.008s
I20260501 13:57:36.059024 1882 ts_tablet_manager.cc:1403] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.140s user 0.009s sys 0.000s
I20260501 13:57:36.061339 1882 raft_consensus.cc:359] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.061764 1130 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 6650472ec4a449c48c7957a9bbec88e3 (table default.Table_0d92e7b95c884bd5970e291f4189b0d9 [id=774a7eb90ba449b6aef225d819678439]). Expected version 1 got 0
I20260501 13:57:36.061970 1882 raft_consensus.cc:385] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.062204 1882 raft_consensus.cc:740] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:36.031288 1778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e86d6c935d6444b8a953e7be1bdfa51" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:36.065502 1778 raft_consensus.cc:3060] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:36.066382 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 31c73582788648aab55d678ad6e4fc76 (DEFAULT_TABLE table=default.Table_c3c929bb31b74edc9cf471501a43baa7 [id=ee3db9bd7d494e5b836198707548c957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:36.069182 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 31c73582788648aab55d678ad6e4fc76 (DEFAULT_TABLE table=default.Table_c3c929bb31b74edc9cf471501a43baa7 [id=ee3db9bd7d494e5b836198707548c957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:36.070341 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c73582788648aab55d678ad6e4fc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:36.073737 1610 tablet_service.cc:1511] Processing CreateTablet for tablet 81daa397384d4afb816071ee665a1dcd (DEFAULT_TABLE table=default.Table_dc19bcebd0934f20b6dc5a1da2f67f21 [id=7ab4b36ba8b8496ea524426efa573d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:36.070214 1882 consensus_queue.cc:260] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.074895 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81daa397384d4afb816071ee665a1dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:36.076125 1882 ts_tablet_manager.cc:1434] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260501 13:57:36.079353 1882 tablet_bootstrap.cc:492] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:36.082278 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e86d6c935d6444b8a953e7be1bdfa51" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:36.083593 1482 raft_consensus.cc:2393] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ed1f9a545474e76a661c32bf11a7fb2 in current term 1: Already voted for candidate 993b9ba18d8547f8af43167e727a6a24 in this term.
I20260501 13:57:36.085161 1717 leader_election.cc:304] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:36.088964 1882 tablet_bootstrap.cc:654] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:36.091815 1778 raft_consensus.cc:2468] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:36.092527 2097 raft_consensus.cc:2749] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:36.093068 2093 tablet_replica.cc:333] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:36.093945 2093 raft_consensus.cc:2243] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:36.094499 2093 raft_consensus.cc:2272] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:36.094324 1418 leader_election.cc:304] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:36.095271 1931 raft_consensus.cc:2804] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:36.095716 1931 raft_consensus.cc:697] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:36.096498 2093 ts_tablet_manager.cc:1916] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:36.096479 1931 consensus_queue.cc:237] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.079707 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c73582788648aab55d678ad6e4fc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:36.022413 1883 tablet_bootstrap.cc:654] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:36.114647 2166 leader_election.cc:290] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:36.123070 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 already in progress: creating tablet
W20260501 13:57:36.134752 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet f32170902c134c93a9aa71b2530c845e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f32170902c134c93a9aa71b2530c845e already in progress: creating tablet
I20260501 13:57:36.061343 1881 raft_consensus.cc:359] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.139540 1881 raft_consensus.cc:385] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.139899 1881 raft_consensus.cc:740] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:36.140633 1881 consensus_queue.cc:260] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.143049 1883 tablet_bootstrap.cc:492] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:36.143494 1883 ts_tablet_manager.cc:1403] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.162s user 0.006s sys 0.003s
I20260501 13:57:36.144383 1131 catalog_manager.cc:5671] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "993b9ba18d8547f8af43167e727a6a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:36.145678 1131 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 8d2f196f44be4fbc8059901380b71271 (table default.Table_360decd489b74f4b8ee93b008e57c45b [id=8cff6f6a2ebc447c979a45c9bfa438dc]). Expected version 2 got 0
I20260501 13:57:36.146418 1131 catalog_manager.cc:5671] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "993b9ba18d8547f8af43167e727a6a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:36.147892 1131 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 5e86d6c935d6444b8a953e7be1bdfa51 (table default.Table_299166301fa440eca9a509c07bde6ab3 [id=0f9c1230865848adaa4e5ea744c31c8b]). Expected version 2 got 0
I20260501 13:57:36.148844 1131 catalog_manager.cc:5671] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "993b9ba18d8547f8af43167e727a6a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:36.149545 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet c5297eb8aace46078cc3fce289fdd389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:36.160202 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 81daa397384d4afb816071ee665a1dcd (DEFAULT_TABLE table=default.Table_dc19bcebd0934f20b6dc5a1da2f67f21 [id=7ab4b36ba8b8496ea524426efa573d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:36.161485 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81daa397384d4afb816071ee665a1dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:36.161669 1881 ts_tablet_manager.cc:1434] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.102s user 0.004s sys 0.001s
I20260501 13:57:36.175758 1899 ts_tablet_manager.cc:1929] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:36.178691 1881 tablet_bootstrap.cc:492] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
W20260501 13:57:36.156992 1123 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50ec3d3f3eb449759a8df52279b2e5e8 consensus state has no leader
I20260501 13:57:36.179319 1131 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 6c2d5336bfec43dbb38ce7071834bca3 (table default.Table_66545ca4c3d540ada57273201e17f2d8 [id=697eaa78d5e541988cd48466c64c5915]). Expected version 1 got 0
I20260501 13:57:36.180306 1899 log.cc:1199] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/9a0d04810f2d465ebe2964797ea30538
I20260501 13:57:36.180964 1899 ts_tablet_manager.cc:1950] T 9a0d04810f2d465ebe2964797ea30538 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:36.176724 1883 raft_consensus.cc:359] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.182870 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 114) took 2717 ms. Trace:
I20260501 13:57:36.182915 1883 raft_consensus.cc:385] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.203310 1883 raft_consensus.cc:740] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:36.204034 1883 consensus_queue.cc:260] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.203418 1899 rpcz_store.cc:276] 0501 13:57:33.465470 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:33.469252 (+ 3782us) service_pool.cc:224] Handling call
0501 13:57:35.915978 (+2446726us) ts_tablet_manager.cc:1151] Deleting tablet 9a0d04810f2d465ebe2964797ea30538
0501 13:57:35.916031 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.111485 (+195454us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.175639 (+ 64154us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.182810 (+ 7171us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2277223}
I20260501 13:57:36.207967 1899 tablet_replica.cc:333] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:36.208742 1899 raft_consensus.cc:2243] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:36.209230 1899 raft_consensus.cc:2272] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:36.209679 1125 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 69626ed095bd4df7859563c176dafffc
I20260501 13:57:36.210947 1125 catalog_manager.cc:6741] ReplaceTablet: tablet 69626ed095bd4df7859563c176dafffc deleted and replaced by tablet 1042bdf58c1f488daeaec284eb3319d5
I20260501 13:57:36.213847 1883 ts_tablet_manager.cc:1434] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.070s user 0.002s sys 0.003s
I20260501 13:57:36.214846 1883 tablet_bootstrap.cc:492] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:36.226940 1899 ts_tablet_manager.cc:1916] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:36.231542 2093 ts_tablet_manager.cc:1929] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260501 13:57:36.232921 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 9a0d04810f2d465ebe2964797ea30538 (table default.Table_a74b32dea54d43a2a8a1b6f2210f38d8 [id=eaa4c83258f94954af2e04eee604f6f0]) successfully deleted
I20260501 13:57:36.236482 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 69626ed095bd4df7859563c176dafffc with delete_type TABLET_DATA_DELETED (replaced by tablet 1042bdf58c1f488daeaec284eb3319d5) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:36.237602 2093 log.cc:1199] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/d5d4064154c744198fa438cd3e3b7288
I20260501 13:57:36.238444 2093 ts_tablet_manager.cc:1950] T d5d4064154c744198fa438cd3e3b7288 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:36.241006 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_427f541417f845efb488354adcc40139 [id=0dc2b91768f84b4d93b18f304d31883e]) successfully deleted
I20260501 13:57:36.251885 1883 tablet_bootstrap.cc:654] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:36.251991 2093 tablet_replica.cc:333] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:36.253043 2093 raft_consensus.cc:2243] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:36.253558 2093 raft_consensus.cc:2272] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:36.253997 1608 tablet_service.cc:1558] Processing DeleteTablet for tablet 69626ed095bd4df7859563c176dafffc with delete_type TABLET_DATA_DELETED (replaced by tablet 1042bdf58c1f488daeaec284eb3319d5) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:36.255172 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 already in progress: creating tablet
W20260501 13:57:36.255927 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet f32170902c134c93a9aa71b2530c845e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f32170902c134c93a9aa71b2530c845e already in progress: creating tablet
W20260501 13:57:36.260080 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e86d6c935d6444b8a953e7be1bdfa51 consensus state has no leader
I20260501 13:57:36.263957 1129 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 2c8d5d04a19443cfaa47b733b1d6ada6
I20260501 13:57:36.267031 1899 ts_tablet_manager.cc:1929] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:36.267634 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 2c8d5d04a19443cfaa47b733b1d6ada6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:36.268138 1899 log.cc:1199] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/1ba63c153ef04db891cae46a71439455
I20260501 13:57:36.268790 1899 ts_tablet_manager.cc:1950] T 1ba63c153ef04db891cae46a71439455 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:36.269086 1754 tablet_service.cc:1511] Processing CreateTablet for tablet 81daa397384d4afb816071ee665a1dcd (DEFAULT_TABLE table=default.Table_dc19bcebd0934f20b6dc5a1da2f67f21 [id=7ab4b36ba8b8496ea524426efa573d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:36.270280 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81daa397384d4afb816071ee665a1dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:36.271533 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 115) took 2784 ms. Trace:
I20260501 13:57:36.280448 1899 rpcz_store.cc:276] 0501 13:57:33.486897 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:33.632959 (+146062us) service_pool.cc:224] Handling call
0501 13:57:36.207810 (+2574851us) ts_tablet_manager.cc:1151] Deleting tablet 1ba63c153ef04db891cae46a71439455
0501 13:57:36.207868 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.235541 (+ 27673us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.266917 (+ 31376us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.271479 (+ 4562us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2538467}
I20260501 13:57:36.281152 1883 tablet_bootstrap.cc:492] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:36.281527 1883 ts_tablet_manager.cc:1403] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.002s
I20260501 13:57:36.282101 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 1ba63c153ef04db891cae46a71439455 (table default.Table_66a4e496efa34deaa6c793a210a510ff [id=e8b19350d702499a82f41a6cce14dbd5]) successfully deleted
I20260501 13:57:36.282372 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 121) took 2203 ms. Trace:
I20260501 13:57:36.282780 1899 rpcz_store.cc:276] 0501 13:57:34.079294 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:34.079975 (+ 681us) service_pool.cc:224] Handling call
0501 13:57:36.281557 (+2201582us) ts_tablet_manager.cc:1151] Deleting tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb
0501 13:57:36.281612 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.282325 (+ 713us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2144418}
I20260501 13:57:36.283773 1899 tablet_replica.cc:333] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
W20260501 13:57:36.284287 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb already in progress: creating tablet
I20260501 13:57:36.285378 1899 raft_consensus.cc:2243] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:36.285825 1899 raft_consensus.cc:2272] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:36.288308 1899 ts_tablet_manager.cc:1916] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:36.305531 1752 tablet_service.cc:1511] Processing CreateTablet for tablet 1606ce24d9654233992fd06222f3ea3e (DEFAULT_TABLE table=default.Table_320db474f1554b269265f3448e635536 [id=0eed8748bd674709b7bb9412b87687fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:36.306038 1883 raft_consensus.cc:359] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.306681 1883 raft_consensus.cc:385] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.306787 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1606ce24d9654233992fd06222f3ea3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:36.306939 1883 raft_consensus.cc:740] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:36.255911 1882 tablet_bootstrap.cc:492] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:36.264588 2093 ts_tablet_manager.cc:1916] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:36.321799 1882 ts_tablet_manager.cc:1403] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.243s user 0.023s sys 0.027s
I20260501 13:57:36.324151 1882 raft_consensus.cc:359] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.324738 1882 raft_consensus.cc:385] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.324972 1882 raft_consensus.cc:740] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:36.325539 1882 consensus_queue.cc:260] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.323239 1883 consensus_queue.cc:260] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.265763 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet 2c8d5d04a19443cfaa47b733b1d6ada6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:36.290975 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 1606ce24d9654233992fd06222f3ea3e (DEFAULT_TABLE table=default.Table_320db474f1554b269265f3448e635536 [id=0eed8748bd674709b7bb9412b87687fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:36.327142 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1606ce24d9654233992fd06222f3ea3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:36.334384 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet 02926865010043e4b689c914b435400f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:36.337642 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_62888a4c57cb4690bf1ab86c99529aa4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:36.339345 1883 ts_tablet_manager.cc:1434] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.058s user 0.003s sys 0.003s
I20260501 13:57:36.345688 1883 tablet_bootstrap.cc:492] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:36.348609 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62888a4c57cb4690bf1ab86c99529aa4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:36.352267 1882 ts_tablet_manager.cc:1434] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.030s user 0.003s sys 0.004s
I20260501 13:57:36.362905 1882 tablet_bootstrap.cc:492] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:36.362730 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_16f2a19c6fb94677ad93beb76d5d9419"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:36.364440 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16f2a19c6fb94677ad93beb76d5d9419 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:36.368546 1882 tablet_bootstrap.cc:654] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:36.383623 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 2c8d5d04a19443cfaa47b733b1d6ada6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:36.384191 1881 tablet_bootstrap.cc:654] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:36.388342 1899 ts_tablet_manager.cc:1929] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:36.388705 1899 log.cc:1199] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/863a10cfd14c4694b7c6141c0851167a
I20260501 13:57:36.389285 1899 ts_tablet_manager.cc:1950] T 863a10cfd14c4694b7c6141c0851167a P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:36.389741 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c7461471f08c462a93d877fb770f1ab9" } new_table_name: "default.Table_6ca908dd49884d579ce8aa66535b611f" modify_external_catalogs: true
I20260501 13:57:36.394307 1883 tablet_bootstrap.cc:654] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:36.398417 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 1606ce24d9654233992fd06222f3ea3e (DEFAULT_TABLE table=default.Table_320db474f1554b269265f3448e635536 [id=0eed8748bd674709b7bb9412b87687fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:36.399878 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1606ce24d9654233992fd06222f3ea3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:36.402778 1463 tablet_service.cc:1558] Processing DeleteTablet for tablet 69626ed095bd4df7859563c176dafffc with delete_type TABLET_DATA_DELETED (replaced by tablet 1042bdf58c1f488daeaec284eb3319d5) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:36.393780 2093 ts_tablet_manager.cc:1929] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:36.403697 2093 log.cc:1199] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/c5297eb8aace46078cc3fce289fdd389
I20260501 13:57:36.404387 2093 ts_tablet_manager.cc:1950] T c5297eb8aace46078cc3fce289fdd389 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:36.404978 1882 tablet_bootstrap.cc:492] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
W20260501 13:57:36.404878 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 2c8d5d04a19443cfaa47b733b1d6ada6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c8d5d04a19443cfaa47b733b1d6ada6 already in progress: creating tablet
I20260501 13:57:36.405388 1882 ts_tablet_manager.cc:1403] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.005s
W20260501 13:57:36.406052 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 69626ed095bd4df7859563c176dafffc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69626ed095bd4df7859563c176dafffc already in progress: creating tablet
I20260501 13:57:36.407747 1882 raft_consensus.cc:359] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.408362 1882 raft_consensus.cc:385] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.408613 1882 raft_consensus.cc:740] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:36.409263 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet c5297eb8aace46078cc3fce289fdd389 (table default.Table_eb039fccfc7d444baf4e6cdddc6be5cd [id=37089fa463db4a608b09504153e2e32a]) successfully deleted
I20260501 13:57:36.410053 1882 consensus_queue.cc:260] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.411691 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 122) took 2299 ms. Trace:
I20260501 13:57:36.412019 1899 rpcz_store.cc:276] 0501 13:57:34.112355 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:34.151703 (+ 39348us) service_pool.cc:224] Handling call
0501 13:57:36.283640 (+2131937us) ts_tablet_manager.cc:1151] Deleting tablet 863a10cfd14c4694b7c6141c0851167a
0501 13:57:36.283681 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.303591 (+ 19910us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.388226 (+ 84635us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.411624 (+ 23398us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2129024}
I20260501 13:57:36.413602 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 863a10cfd14c4694b7c6141c0851167a (table default.Table_95c99739e2574437b16501060ce816af [id=a6953bc50bab4e7f92adfee0ef034465]) successfully deleted
I20260501 13:57:36.417810 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 123) took 2194 ms. Trace:
I20260501 13:57:36.418123 1899 rpcz_store.cc:276] 0501 13:57:34.223129 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:34.223425 (+ 296us) service_pool.cc:224] Handling call
0501 13:57:36.417394 (+2193969us) ts_tablet_manager.cc:1151] Deleting tablet a9672aab85c64d548a4f6583a1e9af20
0501 13:57:36.417462 (+ 68us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.417749 (+ 287us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2192129}
W20260501 13:57:36.419093 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet a9672aab85c64d548a4f6583a1e9af20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9672aab85c64d548a4f6583a1e9af20 already in progress: creating tablet
I20260501 13:57:36.420332 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet adc47b3b2b084d33a12bd28cfad27382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:36.421958 2093 tablet_replica.cc:333] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
W20260501 13:57:36.422112 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 69626ed095bd4df7859563c176dafffc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69626ed095bd4df7859563c176dafffc already in progress: creating tablet
W20260501 13:57:36.422827 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 2c8d5d04a19443cfaa47b733b1d6ada6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c8d5d04a19443cfaa47b733b1d6ada6 already in progress: creating tablet
I20260501 13:57:36.423460 1883 tablet_bootstrap.cc:492] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:36.423862 1883 ts_tablet_manager.cc:1403] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.078s user 0.008s sys 0.001s
I20260501 13:57:36.424328 1882 ts_tablet_manager.cc:1434] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260501 13:57:36.431348 2093 raft_consensus.cc:2243] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:36.431823 2093 raft_consensus.cc:2272] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:36.432935 1882 tablet_bootstrap.cc:492] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:36.430415 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 124) took 2130 ms. Trace:
I20260501 13:57:36.438673 1899 rpcz_store.cc:276] 0501 13:57:34.300091 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:34.300220 (+ 129us) service_pool.cc:224] Handling call
0501 13:57:36.430129 (+2129909us) ts_tablet_manager.cc:1151] Deleting tablet b507a4d4c9c54bc68d92d9286794a264
0501 13:57:36.430188 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.430358 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2129371}
W20260501 13:57:36.439786 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet b507a4d4c9c54bc68d92d9286794a264 because the tablet was not found. No further retry: Not found: Tablet not found: b507a4d4c9c54bc68d92d9286794a264
I20260501 13:57:36.441111 1883 raft_consensus.cc:359] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.442499 2093 ts_tablet_manager.cc:1916] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:36.444849 1882 tablet_bootstrap.cc:654] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:36.443392 1899 tablet_replica.cc:333] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:36.447206 1883 raft_consensus.cc:385] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.450639 1883 raft_consensus.cc:740] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:36.447614 1899 raft_consensus.cc:2243] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:36.453166 1899 raft_consensus.cc:2272] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:36.455112 1899 ts_tablet_manager.cc:1916] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:36.464308 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6fe081b50a0640afbbb524ff85b821d2" } new_table_name: "default.Table_cf75782d380e49e3b4cf46f49c816974" modify_external_catalogs: true
I20260501 13:57:36.453222 1883 consensus_queue.cc:260] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.466437 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7bbaf820afc746198b26a791c87cb736"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:36.470162 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bbaf820afc746198b26a791c87cb736 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:36.472932 1883 ts_tablet_manager.cc:1434] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.049s user 0.003s sys 0.003s
I20260501 13:57:36.475948 1123 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_5be69bddaf9d4f6a8e87137a8a46e223" } modify_external_catalogs: true
I20260501 13:57:36.476547 1123 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_5be69bddaf9d4f6a8e87137a8a46e223" } modify_external_catalogs: true
I20260501 13:57:36.486923 1883 tablet_bootstrap.cc:492] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:36.487097 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet d361a2d82ecd456fb43f5e7fe3c83094
I20260501 13:57:36.489252 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet d361a2d82ecd456fb43f5e7fe3c83094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:36.489478 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet d361a2d82ecd456fb43f5e7fe3c83094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:36.494252 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_68ae6dc027ee4521baf488d0c406a1e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:36.495975 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68ae6dc027ee4521baf488d0c406a1e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:36.523898 2108 raft_consensus.cc:493] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:36.524494 2108 raft_consensus.cc:515] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.524906 2093 ts_tablet_manager.cc:1929] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:36.525264 2093 log.cc:1199] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/02926865010043e4b689c914b435400f
I20260501 13:57:36.525869 2093 ts_tablet_manager.cc:1950] T 02926865010043e4b689c914b435400f P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:36.528075 2093 tablet_replica.cc:333] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:36.528731 2108 leader_election.cc:290] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:36.529189 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 02926865010043e4b689c914b435400f (table default.Table_4bb39ebdce2145ccaf94f17bb84197b8 [id=a7c0d6e58eac4f9aabd0f2bd585db9a6]) successfully deleted
I20260501 13:57:36.529721 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c2d5336bfec43dbb38ce7071834bca3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:36.544310 1899 ts_tablet_manager.cc:1929] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:36.544734 1899 log.cc:1199] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/eeae4369be954d9d9bad846484afb6ff
I20260501 13:57:36.545519 1899 ts_tablet_manager.cc:1950] T eeae4369be954d9d9bad846484afb6ff P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:36.547580 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 125) took 1897 ms. Trace:
I20260501 13:57:36.547886 1899 rpcz_store.cc:276] 0501 13:57:34.650265 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:34.650384 (+ 119us) service_pool.cc:224] Handling call
0501 13:57:36.443233 (+1792849us) ts_tablet_manager.cc:1151] Deleting tablet eeae4369be954d9d9bad846484afb6ff
0501 13:57:36.443287 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.499490 (+ 56203us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.544151 (+ 44661us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.547505 (+ 3354us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1792239}
I20260501 13:57:36.549692 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet eeae4369be954d9d9bad846484afb6ff (table default.Table_6452fc6a9103447c9239a13e344f0c5a [id=ecd108cb5a5544c590e904bc9185a0d0]) successfully deleted
I20260501 13:57:36.561393 1883 tablet_bootstrap.cc:654] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:36.565744 1719 consensus_peers.cc:597] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5d4064154c744198fa438cd3e3b7288. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:36.570659 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 126) took 1849 ms. Trace:
I20260501 13:57:36.571007 1899 rpcz_store.cc:276] 0501 13:57:34.721495 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:34.721616 (+ 121us) service_pool.cc:224] Handling call
0501 13:57:36.570244 (+1848628us) ts_tablet_manager.cc:1151] Deleting tablet e1d7a80ad4c4455a9ddee7a955e32635
0501 13:57:36.570304 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.570603 (+ 299us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1848102}
W20260501 13:57:36.572072 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet e1d7a80ad4c4455a9ddee7a955e32635 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1d7a80ad4c4455a9ddee7a955e32635 already in progress: creating tablet
I20260501 13:57:36.579216 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 129) took 1714 ms. Trace:
I20260501 13:57:36.579564 1899 rpcz_store.cc:276] 0501 13:57:34.864629 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:34.918187 (+ 53558us) service_pool.cc:224] Handling call
0501 13:57:36.578791 (+1660604us) ts_tablet_manager.cc:1151] Deleting tablet eaf868a5d2c24a45b183299d71f082a9
0501 13:57:36.578839 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.579147 (+ 308us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1659837}
W20260501 13:57:36.580619 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet eaf868a5d2c24a45b183299d71f082a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eaf868a5d2c24a45b183299d71f082a9 already in progress: creating tablet
I20260501 13:57:36.584750 1883 tablet_bootstrap.cc:492] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:36.587080 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 130) took 1722 ms. Trace:
I20260501 13:57:36.590756 1883 ts_tablet_manager.cc:1403] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.104s user 0.008s sys 0.005s
I20260501 13:57:36.590217 1899 rpcz_store.cc:276] 0501 13:57:34.864931 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:34.921602 (+ 56671us) service_pool.cc:224] Handling call
0501 13:57:36.586715 (+1665113us) ts_tablet_manager.cc:1151] Deleting tablet ec4bd4cb1e70472aa09fc93afc3b1501
0501 13:57:36.586756 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.587007 (+ 251us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1664554}
W20260501 13:57:36.594914 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet ec4bd4cb1e70472aa09fc93afc3b1501 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec4bd4cb1e70472aa09fc93afc3b1501 already in progress: creating tablet
I20260501 13:57:36.595337 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 139) took 1422 ms. Trace:
I20260501 13:57:36.597540 1883 raft_consensus.cc:359] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.602913 1883 raft_consensus.cc:385] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.595647 1899 rpcz_store.cc:276] 0501 13:57:35.173007 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:35.173142 (+ 135us) service_pool.cc:224] Handling call
0501 13:57:36.594975 (+1421833us) ts_tablet_manager.cc:1151] Deleting tablet 39cd02738e5f4ecfbfa06d5541614ffd
0501 13:57:36.595019 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.595278 (+ 259us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1410321}
W20260501 13:57:36.604437 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 39cd02738e5f4ecfbfa06d5541614ffd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39cd02738e5f4ecfbfa06d5541614ffd already in progress: creating tablet
I20260501 13:57:36.604671 1899 tablet_replica.cc:333] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:36.603840 1883 raft_consensus.cc:740] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:36.605336 1899 raft_consensus.cc:2243] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:36.605799 1778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c2d5336bfec43dbb38ce7071834bca3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:36.606640 1778 raft_consensus.cc:2393] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f90b7ff170b4eadb27abc9ef866ffcf in current term 1: Already voted for candidate 993b9ba18d8547f8af43167e727a6a24 in this term.
I20260501 13:57:36.605968 1883 consensus_queue.cc:260] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.605792 1899 pending_rounds.cc:70] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: Trying to abort 1 pending ops.
I20260501 13:57:36.610349 1899 pending_rounds.cc:77] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229228496216064 op_type: NO_OP noop_request { }
I20260501 13:57:36.610785 1899 raft_consensus.cc:2889] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:36.611426 2097 raft_consensus.cc:493] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:36.611832 2097 raft_consensus.cc:515] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.612321 1882 tablet_bootstrap.cc:492] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:36.612775 1882 ts_tablet_manager.cc:1403] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.180s user 0.013s sys 0.033s
I20260501 13:57:36.613629 2097 leader_election.cc:290] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:36.615231 1882 raft_consensus.cc:359] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.615850 1882 raft_consensus.cc:385] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.615718 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c33a1f423bdb44f5843daffe8f73e74a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:36.616142 1882 raft_consensus.cc:740] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
W20260501 13:57:36.617053 1719 leader_election.cc:343] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.620097 1563 leader_election.cc:304] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:36.621201 2108 raft_consensus.cc:3060] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:36.611105 1899 raft_consensus.cc:2272] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:36.623989 1899 ts_tablet_manager.cc:1916] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:36.626736 1882 consensus_queue.cc:260] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.628208 2108 raft_consensus.cc:2749] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:36.627220 1883 ts_tablet_manager.cc:1434] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.036s user 0.005s sys 0.002s
I20260501 13:57:36.630409 1883 tablet_bootstrap.cc:492] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:36.631910 2097 raft_consensus.cc:493] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:36.632344 2097 raft_consensus.cc:515] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.634217 2097 leader_election.cc:290] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:36.638571 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe89d656956440e8d0c1de0b01ce486" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:36.644711 1719 leader_election.cc:343] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.649959 1883 tablet_bootstrap.cc:654] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:36.656292 1124 master_service.cc:954] ReplaceTablet: received request to replace tablet eeae4369be954d9d9bad846484afb6ff from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:36.679071 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c33a1f423bdb44f5843daffe8f73e74a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:36.681732 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet d361a2d82ecd456fb43f5e7fe3c83094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:36.684803 1931 raft_consensus.cc:493] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:36.685272 1931 raft_consensus.cc:515] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.687090 1931 leader_election.cc:290] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:36.693286 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ffe89d656956440e8d0c1de0b01ce486 consensus state has no leader
I20260501 13:57:36.680104 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe89d656956440e8d0c1de0b01ce486" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:36.709561 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a563bccc7b046658ed133109f676528" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:36.710191 1629 raft_consensus.cc:2468] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:36.712682 1883 tablet_bootstrap.cc:492] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:36.712621 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ddd647c4319b431f9d4e874bb528755d" } modify_external_catalogs: true
I20260501 13:57:36.713079 1883 ts_tablet_manager.cc:1403] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.083s user 0.013s sys 0.008s
I20260501 13:57:36.713160 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ddd647c4319b431f9d4e874bb528755d" } modify_external_catalogs: true
I20260501 13:57:36.715366 1883 raft_consensus.cc:359] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.715966 1883 raft_consensus.cc:385] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.716187 1883 raft_consensus.cc:740] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
W20260501 13:57:36.718636 1717 consensus_peers.cc:597] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5d4064154c744198fa438cd3e3b7288. This is attempt 1: this message will repeat every 5th retry.
W20260501 13:57:36.719638 1717 leader_election.cc:343] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.720201 1717 leader_election.cc:304] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:36.721450 1717 leader_election.cc:343] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.724499 2097 raft_consensus.cc:2749] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:36.726644 1421 leader_election.cc:304] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:36.727823 1883 consensus_queue.cc:260] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.729264 1931 raft_consensus.cc:2804] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:36.729707 1883 ts_tablet_manager.cc:1434] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260501 13:57:36.729677 1931 raft_consensus.cc:493] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:36.730005 1931 raft_consensus.cc:3060] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:36.731545 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet d361a2d82ecd456fb43f5e7fe3c83094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d361a2d82ecd456fb43f5e7fe3c83094 already in progress: creating tablet
I20260501 13:57:36.731803 1899 ts_tablet_manager.cc:1929] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:36.742995 1883 tablet_bootstrap.cc:492] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:36.750552 1899 log.cc:1199] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/c5297eb8aace46078cc3fce289fdd389
I20260501 13:57:36.751343 1899 ts_tablet_manager.cc:1950] T c5297eb8aace46078cc3fce289fdd389 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:36.721866 1717 leader_election.cc:304] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:36.753397 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 140) took 1222 ms. Trace:
I20260501 13:57:36.755820 1899 rpcz_store.cc:276] 0501 13:57:35.530865 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:35.531276 (+ 411us) service_pool.cc:224] Handling call
0501 13:57:36.604521 (+1073245us) ts_tablet_manager.cc:1151] Deleting tablet c5297eb8aace46078cc3fce289fdd389
0501 13:57:36.604564 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.676640 (+ 72076us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.731678 (+ 55038us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.753335 (+ 21657us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1072515}
I20260501 13:57:36.756317 2097 raft_consensus.cc:2749] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:36.756969 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet c5297eb8aace46078cc3fce289fdd389 (table default.Table_eb039fccfc7d444baf4e6cdddc6be5cd [id=37089fa463db4a608b09504153e2e32a]) successfully deleted
I20260501 13:57:36.757232 1899 tablet_replica.cc:333] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:36.749010 2166 consensus_queue.cc:1048] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:57:36.757897 1899 raft_consensus.cc:2243] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:36.763325 1899 raft_consensus.cc:2272] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:36.763633 2097 raft_consensus.cc:493] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:36.764140 2097 raft_consensus.cc:515] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.765295 1899 ts_tablet_manager.cc:1916] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:36.767151 2097 leader_election.cc:290] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:36.768827 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f2448ee6bf14bbc8ec0e1d59ccb7b90" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:36.769784 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f2448ee6bf14bbc8ec0e1d59ccb7b90" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:36.770267 1719 leader_election.cc:343] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.772629 2166 consensus_queue.cc:1048] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:36.776424 1717 leader_election.cc:343] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.776898 1717 leader_election.cc:304] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:36.781589 2097 raft_consensus.cc:2749] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:36.781142 1883 tablet_bootstrap.cc:654] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:36.796698 1883 tablet_bootstrap.cc:492] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:36.796861 1881 tablet_bootstrap.cc:492] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:36.797364 1881 ts_tablet_manager.cc:1403] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.619s user 0.007s sys 0.007s
I20260501 13:57:36.799540 1883 ts_tablet_manager.cc:1403] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.000s
W20260501 13:57:36.804293 1421 consensus_peers.cc:597] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:36.809845 1883 raft_consensus.cc:359] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.811887 1883 raft_consensus.cc:385] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.812552 1883 raft_consensus.cc:740] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:36.813531 1883 consensus_queue.cc:260] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.822065 2207 raft_consensus.cc:493] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:36.826540 1881 raft_consensus.cc:359] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.827246 1881 raft_consensus.cc:385] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.823712 2097 raft_consensus.cc:493] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:36.829241 2097 raft_consensus.cc:515] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.832715 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9018d70c734a429aa59565ad67a46acc" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:36.820951 1883 ts_tablet_manager.cc:1434] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.021s user 0.004s sys 0.001s
W20260501 13:57:36.834090 1719 leader_election.cc:343] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.838634 2097 leader_election.cc:290] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:36.840143 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9018d70c734a429aa59565ad67a46acc" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:36.841683 1717 leader_election.cc:343] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.842128 1717 leader_election.cc:304] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:36.844889 2207 raft_consensus.cc:515] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.847756 2097 raft_consensus.cc:2749] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:36.847750 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9672aab85c64d548a4f6583a1e9af20" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:36.848188 1899 ts_tablet_manager.cc:1929] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260501 13:57:36.848529 1899 log.cc:1199] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/d5d4064154c744198fa438cd3e3b7288
I20260501 13:57:36.848932 1883 tablet_bootstrap.cc:492] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:36.849222 1899 ts_tablet_manager.cc:1950] T d5d4064154c744198fa438cd3e3b7288 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
W20260501 13:57:36.849475 1719 leader_election.cc:343] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.858204 2097 raft_consensus.cc:493] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:36.858819 2097 raft_consensus.cc:515] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.862483 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6650472ec4a449c48c7957a9bbec88e3" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:36.863945 1719 leader_election.cc:343] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.865752 2097 leader_election.cc:290] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:36.866014 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6650472ec4a449c48c7957a9bbec88e3" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:36.867563 1717 leader_election.cc:343] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.528781 2093 raft_consensus.cc:2243] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:36.867985 1717 leader_election.cc:304] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:36.868629 2093 raft_consensus.cc:2272] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:36.868983 2097 raft_consensus.cc:2749] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:36.869985 2108 raft_consensus.cc:670] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20260501 13:57:36.870450 1883 tablet_bootstrap.cc:654] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:36.876747 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 142) took 1209 ms. Trace:
I20260501 13:57:36.877179 1899 rpcz_store.cc:276] 0501 13:57:35.667534 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:35.667646 (+ 112us) service_pool.cc:224] Handling call
0501 13:57:36.757094 (+1089448us) ts_tablet_manager.cc:1151] Deleting tablet d5d4064154c744198fa438cd3e3b7288
0501 13:57:36.757133 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:36.812265 (+ 55132us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.848075 (+ 35810us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:36.876664 (+ 28589us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4547,"tablet-delete.queue_time_us":1088908}
I20260501 13:57:36.878558 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet d5d4064154c744198fa438cd3e3b7288 (table default.Table_427f541417f845efb488354adcc40139 [id=0dc2b91768f84b4d93b18f304d31883e]) successfully deleted
I20260501 13:57:36.882877 2207 leader_election.cc:290] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:36.891602 1899 tablet_replica.cc:333] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:36.891738 1883 tablet_bootstrap.cc:492] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:36.893241 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet d5d4064154c744198fa438cd3e3b7288 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:36.892994 1883 ts_tablet_manager.cc:1403] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.000s
I20260501 13:57:36.894997 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_d23b951fef3048af97eb045b842ebd22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:36.892385 1899 raft_consensus.cc:2243] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:36.896271 1883 raft_consensus.cc:359] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.886794 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9672aab85c64d548a4f6583a1e9af20" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:36.896878 1883 raft_consensus.cc:385] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:36.897150 1883 raft_consensus.cc:740] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
W20260501 13:57:36.897812 1717 leader_election.cc:343] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.897738 1883 consensus_queue.cc:260] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.898268 1717 leader_election.cc:304] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:36.912262 2097 raft_consensus.cc:2749] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:36.912478 1883 ts_tablet_manager.cc:1434] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.018s user 0.003s sys 0.001s
W20260501 13:57:36.915356 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d23b951fef3048af97eb045b842ebd22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:36.915622 1883 tablet_bootstrap.cc:492] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:36.917673 1899 raft_consensus.cc:2272] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:36.896955 1128 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50ec3d3f3eb449759a8df52279b2e5e8 consensus state has no leader
I20260501 13:57:36.897213 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_7591dec5d3bf43b8b63db3443295ed26" } new_table_name: "default.Table_ad4c78cb126f4013968e7b0e20a5c47c" modify_external_catalogs: true
I20260501 13:57:36.933504 2166 consensus_queue.cc:1048] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:36.936273 2166 consensus_queue.cc:1048] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:36.937039 1883 tablet_bootstrap.cc:654] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:36.938159 1117 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 50ec3d3f3eb449759a8df52279b2e5e8 (table default.Table_cf75782d380e49e3b4cf46f49c816974 [id=32fd27a2bdec45698663dac24dd99b12], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 50ec3d3f3eb449759a8df52279b2e5e8 consensus state has no leader
W20260501 13:57:36.938418 1418 leader_election.cc:343] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 4a563bccc7b046658ed133109f676528
I20260501 13:57:36.943876 2166 consensus_queue.cc:1048] T f7645593dc184dab9f677f9371a88354 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:36.946915 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_d132a7668b34479887548c901967b36c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:36.950886 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d132a7668b34479887548c901967b36c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:36.956422 2097 raft_consensus.cc:493] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:36.956904 2097 raft_consensus.cc:515] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.959110 2097 leader_election.cc:290] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:36.959479 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b98187c7843e42b09d55c851117c4ac8" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:36.960796 1719 leader_election.cc:343] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.963047 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b98187c7843e42b09d55c851117c4ac8" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:36.965854 1717 leader_election.cc:343] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:36.966289 1717 leader_election.cc:304] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:36.966991 2206 raft_consensus.cc:2749] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:36.974503 1119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20260501 13:57:36.827499 1881 raft_consensus.cc:740] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:36.905426 1123 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet e0356b19f0aa4197a41cc66d4440bffb
I20260501 13:57:36.856433 1931 raft_consensus.cc:515] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.989972 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet e0356b19f0aa4197a41cc66d4440bffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:36.993592 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4e1547b16dcc4d878bcc141802b827dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:36.995312 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e1547b16dcc4d878bcc141802b827dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:36.996179 1881 consensus_queue.cc:260] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.999894 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a563bccc7b046658ed133109f676528" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:37.000465 1629 raft_consensus.cc:3060] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:37.006695 1899 ts_tablet_manager.cc:1916] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:37.008996 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet e0356b19f0aa4197a41cc66d4440bffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:37.010278 1629 raft_consensus.cc:2468] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
W20260501 13:57:37.010511 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet e0356b19f0aa4197a41cc66d4440bffb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0356b19f0aa4197a41cc66d4440bffb already in progress: creating tablet
I20260501 13:57:37.015470 1421 leader_election.cc:304] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:37.016371 2166 raft_consensus.cc:2804] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:37.016870 2166 raft_consensus.cc:697] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:37.021781 2166 consensus_queue.cc:237] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.035470 2093 ts_tablet_manager.cc:1916] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:37.036993 1882 ts_tablet_manager.cc:1434] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.419s user 0.035s sys 0.077s
I20260501 13:57:37.047452 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet e0356b19f0aa4197a41cc66d4440bffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:37.056355 1931 leader_election.cc:290] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:37.089740 1883 tablet_bootstrap.cc:492] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:37.090248 1883 ts_tablet_manager.cc:1403] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.175s user 0.014s sys 0.033s
W20260501 13:57:37.092558 1421 consensus_peers.cc:597] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:37.093861 1882 tablet_bootstrap.cc:492] T f7645593dc184dab9f677f9371a88354 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:37.105967 1882 tablet_bootstrap.cc:654] T f7645593dc184dab9f677f9371a88354 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.119467 1883 raft_consensus.cc:359] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.120113 1883 raft_consensus.cc:385] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.120368 1883 raft_consensus.cc:740] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.120965 1883 consensus_queue.cc:260] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:36.998365 1881 ts_tablet_manager.cc:1434] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.201s user 0.006s sys 0.001s
I20260501 13:57:37.127178 1881 tablet_bootstrap.cc:492] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:37.136722 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_761ae80bcd5e4b1ebde69f67afb292eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:37.137722 1130 master_service.cc:954] ReplaceTablet: received request to replace tablet 670f40234b8f43d485027818ac9694c8 from {username='slave'} at 127.0.0.1:41700
W20260501 13:57:37.138252 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_761ae80bcd5e4b1ebde69f67afb292eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:37.142055 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_89c5b57152224983bcb4b5ef5acd7af4" } new_table_name: "default.Table_94e9163950b1491b8dbba021591c6123" modify_external_catalogs: true
W20260501 13:57:37.149351 1418 leader_election.cc:343] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 4a563bccc7b046658ed133109f676528
I20260501 13:57:37.166375 1883 ts_tablet_manager.cc:1434] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.076s user 0.005s sys 0.001s
I20260501 13:57:37.167389 1883 tablet_bootstrap.cc:492] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:37.178730 1125 master_service.cc:954] ReplaceTablet: received request to replace tablet 90a57bc58e854e469897a04b2d434d06 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:37.183544 1899 ts_tablet_manager.cc:1929] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:37.183945 1899 log.cc:1199] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/4f2448ee6bf14bbc8ec0e1d59ccb7b90
I20260501 13:57:37.184554 1899 ts_tablet_manager.cc:1950] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:37.187501 2242 tablet.cc:1724] T 6c2d5336bfec43dbb38ce7071834bca3 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 13:57:37.189103 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8efee17e2c34caf98c0ede01d2a943a consensus state has no leader
I20260501 13:57:37.191411 1882 tablet_bootstrap.cc:492] T f7645593dc184dab9f677f9371a88354 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:37.191820 1882 ts_tablet_manager.cc:1403] T f7645593dc184dab9f677f9371a88354 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.154s user 0.011s sys 0.001s
I20260501 13:57:37.192584 2243 tablet.cc:1724] T 6c2d5336bfec43dbb38ce7071834bca3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:37.198843 1130 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 670f40234b8f43d485027818ac9694c8
I20260501 13:57:37.200364 1456 tablet_service.cc:1558] Processing DeleteTablet for tablet 670f40234b8f43d485027818ac9694c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 069f9e9b9b6c40d48c9a622b73e83dec) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:37.200757 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet 670f40234b8f43d485027818ac9694c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 069f9e9b9b6c40d48c9a622b73e83dec) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:37.201324 1130 catalog_manager.cc:6741] ReplaceTablet: tablet 670f40234b8f43d485027818ac9694c8 deleted and replaced by tablet 069f9e9b9b6c40d48c9a622b73e83dec
I20260501 13:57:37.201777 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 670f40234b8f43d485027818ac9694c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 069f9e9b9b6c40d48c9a622b73e83dec) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:37.186376 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 146) took 1371 ms. Trace:
I20260501 13:57:37.206562 1899 rpcz_store.cc:276] 0501 13:57:35.815239 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:35.815365 (+ 126us) service_pool.cc:224] Handling call
0501 13:57:36.891435 (+1076070us) ts_tablet_manager.cc:1151] Deleting tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90
0501 13:57:36.891486 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:37.152291 (+260805us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:37.183415 (+ 31124us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:37.186320 (+ 2905us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6827,"tablet-delete.queue_time_us":1050263}
I20260501 13:57:37.207813 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 (table default.Table_041fce4b819d4a4c80c58137b4213234 [id=797990ef44df4a5685fa58bd4c3547bd]) successfully deleted
I20260501 13:57:37.209427 2047 tablet_replica.cc:333] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:37.210150 2047 raft_consensus.cc:2243] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:37.210505 1883 tablet_bootstrap.cc:654] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.210662 2047 raft_consensus.cc:2272] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:37.211823 1881 tablet_bootstrap.cc:654] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.212469 2047 ts_tablet_manager.cc:1916] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:37.194191 1882 raft_consensus.cc:359] T f7645593dc184dab9f677f9371a88354 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.213306 1882 raft_consensus.cc:385] T f7645593dc184dab9f677f9371a88354 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.213572 1882 raft_consensus.cc:740] T f7645593dc184dab9f677f9371a88354 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.215251 2232 consensus_queue.cc:1048] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:57:37.217820 1882 consensus_queue.cc:260] T f7645593dc184dab9f677f9371a88354 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.239378 1882 ts_tablet_manager.cc:1434] T f7645593dc184dab9f677f9371a88354 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.047s user 0.007s sys 0.000s
I20260501 13:57:37.242432 1882 tablet_bootstrap.cc:492] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:37.248378 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 150) took 1271 ms. Trace:
I20260501 13:57:37.249974 1130 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_0c2d991875174eefbc2fa69108cc7082" } modify_external_catalogs: true
I20260501 13:57:37.250545 1130 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_0c2d991875174eefbc2fa69108cc7082" } modify_external_catalogs: true
I20260501 13:57:37.254823 1605 tablet_service.cc:1511] Processing CreateTablet for tablet fd5d1afb88d14ddeb09a1c4d28324a35 (DEFAULT_TABLE table=default.Table_200f07afb8cf4eafbace88f5cd3090d2 [id=5c600e77298a4757a2801c398370d5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.255920 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd5d1afb88d14ddeb09a1c4d28324a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.258541 1610 tablet_service.cc:1511] Processing CreateTablet for tablet 5a7eecc7345949cc81ad73e2d0370c9c (DEFAULT_TABLE table=default.Table_0abda464a00b44f49271dbde4c90e526 [id=3d3c9bce9beb4ed5b2614726e6886aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.258780 1882 tablet_bootstrap.cc:654] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.259465 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a7eecc7345949cc81ad73e2d0370c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.268965 1456 tablet_service.cc:1511] Processing CreateTablet for tablet fd5d1afb88d14ddeb09a1c4d28324a35 (DEFAULT_TABLE table=default.Table_200f07afb8cf4eafbace88f5cd3090d2 [id=5c600e77298a4757a2801c398370d5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.270054 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd5d1afb88d14ddeb09a1c4d28324a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.275128 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 5a7eecc7345949cc81ad73e2d0370c9c (DEFAULT_TABLE table=default.Table_0abda464a00b44f49271dbde4c90e526 [id=3d3c9bce9beb4ed5b2614726e6886aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.276068 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a7eecc7345949cc81ad73e2d0370c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.276592 1882 tablet_bootstrap.cc:492] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:37.277040 1882 ts_tablet_manager.cc:1403] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.001s
I20260501 13:57:37.278952 2093 ts_tablet_manager.cc:1929] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:37.285858 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_1d160b6826e74a69979a8b8500b8ca06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:37.287344 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d160b6826e74a69979a8b8500b8ca06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:37.290249 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_6bab24ce4e874b7abcaf569c239b41ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:37.291738 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bab24ce4e874b7abcaf569c239b41ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:37.291950 1132 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715]). Expected version 1 got 0
I20260501 13:57:37.294147 1129 catalog_manager.cc:5671] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "993b9ba18d8547f8af43167e727a6a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:37.295480 1129 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 5e86d6c935d6444b8a953e7be1bdfa51 (table default.Table_7591dec5d3bf43b8b63db3443295ed26 [id=0f9c1230865848adaa4e5ea744c31c8b]). Expected version 3 got 0
W20260501 13:57:37.296486 1126 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e86d6c935d6444b8a953e7be1bdfa51 consensus state has no leader
I20260501 13:57:37.297662 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_d04310990233457bbf23e88ca7321db7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:37.299237 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d04310990233457bbf23e88ca7321db7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:37.302351 1463 tablet_service.cc:1511] Processing CreateTablet for tablet b125d1a190f947a5b2ef942f04408c51 (DEFAULT_TABLE table=default.Table_db50865558a14081b6f0280d9f45ceea [id=d9b238f8c49e4842b7055439df41703d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.303496 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b125d1a190f947a5b2ef942f04408c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.307780 1602 tablet_service.cc:1511] Processing CreateTablet for tablet b125d1a190f947a5b2ef942f04408c51 (DEFAULT_TABLE table=default.Table_db50865558a14081b6f0280d9f45ceea [id=d9b238f8c49e4842b7055439df41703d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.321736 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b125d1a190f947a5b2ef942f04408c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.308911 1608 tablet_service.cc:1511] Processing CreateTablet for tablet 0946834fac5f48a6a9976c21a45e0044 (DEFAULT_TABLE table=default.Table_65c41fbebfea4a3c8bacf216d517e869 [id=909a6cae06f54824bd1f573a8dc8e4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.327389 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0946834fac5f48a6a9976c21a45e0044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.333614 1882 raft_consensus.cc:359] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.334251 1882 raft_consensus.cc:385] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.346556 1882 raft_consensus.cc:740] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.347244 1882 consensus_queue.cc:260] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.352216 1882 ts_tablet_manager.cc:1434] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.075s user 0.006s sys 0.000s
I20260501 13:57:37.355356 1882 tablet_bootstrap.cc:492] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:37.357344 2093 log.cc:1199] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/adc47b3b2b084d33a12bd28cfad27382
I20260501 13:57:37.359071 2093 ts_tablet_manager.cc:1950] T adc47b3b2b084d33a12bd28cfad27382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:37.362732 2093 tablet_replica.cc:333] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:37.363448 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet adc47b3b2b084d33a12bd28cfad27382 (table default.Table_c4edbd4ba4c54b1e9f9909c9badd7c54 [id=1de866dfd0d145e7a13e5ae1f367524b]) successfully deleted
W20260501 13:57:37.364130 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet d361a2d82ecd456fb43f5e7fe3c83094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d361a2d82ecd456fb43f5e7fe3c83094 already in progress: creating tablet
W20260501 13:57:37.364699 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet e0356b19f0aa4197a41cc66d4440bffb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0356b19f0aa4197a41cc66d4440bffb already in progress: creating tablet
I20260501 13:57:37.373670 1463 tablet_service.cc:1511] Processing CreateTablet for tablet 0946834fac5f48a6a9976c21a45e0044 (DEFAULT_TABLE table=default.Table_65c41fbebfea4a3c8bacf216d517e869 [id=909a6cae06f54824bd1f573a8dc8e4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.374752 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0946834fac5f48a6a9976c21a45e0044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.251284 2248 tablet.cc:1724] T 6c2d5336bfec43dbb38ce7071834bca3 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:37.279346 2207 raft_consensus.cc:493] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:37.384096 2207 raft_consensus.cc:515] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.329774 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:37.248812 1899 rpcz_store.cc:276] 0501 13:57:35.977228 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:35.977362 (+ 134us) service_pool.cc:224] Handling call
0501 13:57:37.247974 (+1270612us) ts_tablet_manager.cc:1151] Deleting tablet f32170902c134c93a9aa71b2530c845e
0501 13:57:37.248018 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:37.248322 (+ 304us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1270052}
W20260501 13:57:37.387672 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet f32170902c134c93a9aa71b2530c845e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f32170902c134c93a9aa71b2530c845e already in progress: creating tablet
I20260501 13:57:37.398413 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_3689c91dc5a941d0a884485d780a9ef4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:37.399986 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3689c91dc5a941d0a884485d780a9ef4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:37.408277 1131 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.130:35289 (request call id 59) took 761 ms (client timeout 1000 ms). Trace:
W20260501 13:57:37.408654 1131 rpcz_store.cc:269] 0501 13:57:36.646674 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:36.905748 (+259074us) service_pool.cc:224] Handling call
0501 13:57:37.408218 (+502470us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260501 13:57:37.409948 1463 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a563bccc7b046658ed133109f676528 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:37.411312 1127 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 67b0675398d04be08bd29638b4ba2083
I20260501 13:57:37.412845 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4915b393002a47e895bfa0e9cd82da89" } new_table_name: "default.Table_220e55459a0a41c69abf3fd74bca7bdd" modify_external_catalogs: true
I20260501 13:57:37.414693 1463 tablet_service.cc:1558] Processing DeleteTablet for tablet 67b0675398d04be08bd29638b4ba2083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:37.425720 1127 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 5e86d6c935d6444b8a953e7be1bdfa51 (table default.Table_ad4c78cb126f4013968e7b0e20a5c47c [id=0f9c1230865848adaa4e5ea744c31c8b]). Expected version 4 got 0
I20260501 13:57:37.426992 1125 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 90a57bc58e854e469897a04b2d434d06
I20260501 13:57:37.430780 1125 catalog_manager.cc:6741] ReplaceTablet: tablet 90a57bc58e854e469897a04b2d434d06 deleted and replaced by tablet 3223417aa5594ef7942c6d90bdc8711c
I20260501 13:57:37.433271 1127 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 8d2f196f44be4fbc8059901380b71271 (table default.Table_360decd489b74f4b8ee93b008e57c45b [id=8cff6f6a2ebc447c979a45c9bfa438dc]). Expected version 2 got 0
I20260501 13:57:37.435945 1881 tablet_bootstrap.cc:492] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:37.436349 1881 ts_tablet_manager.cc:1403] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.309s user 0.013s sys 0.003s
I20260501 13:57:37.440075 1463 tablet_service.cc:1511] Processing CreateTablet for tablet 4b3ac1f61eaf461384ae6a956ab56e62 (DEFAULT_TABLE table=default.Table_907a34a78d554ca8a54102bcf65f7ed7 [id=65f926c9bcbd48fe898fe450f021dae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.441151 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b3ac1f61eaf461384ae6a956ab56e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.444456 1461 tablet_service.cc:1511] Processing CreateTablet for tablet a2b292910bf0464cb88efe70b5b0b730 (DEFAULT_TABLE table=default.Table_7127557407564f3b824f44c92babdae3 [id=aa427a761f614ba8b607fc3cc39281a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.445360 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b292910bf0464cb88efe70b5b0b730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.369268 1882 tablet_bootstrap.cc:654] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.376518 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 4b3ac1f61eaf461384ae6a956ab56e62 (DEFAULT_TABLE table=default.Table_907a34a78d554ca8a54102bcf65f7ed7 [id=65f926c9bcbd48fe898fe450f021dae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.416570 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a563bccc7b046658ed133109f676528 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:37.418418 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 67b0675398d04be08bd29638b4ba2083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:37.447711 2093 raft_consensus.cc:2243] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:37.448570 2093 raft_consensus.cc:2272] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:37.432874 1607 tablet_service.cc:1558] Processing DeleteTablet for tablet 90a57bc58e854e469897a04b2d434d06 with delete_type TABLET_DATA_DELETED (replaced by tablet 3223417aa5594ef7942c6d90bdc8711c) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:37.443238 1603 tablet_service.cc:1511] Processing CreateTablet for tablet a2b292910bf0464cb88efe70b5b0b730 (DEFAULT_TABLE table=default.Table_7127557407564f3b824f44c92babdae3 [id=aa427a761f614ba8b607fc3cc39281a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.454277 1125 master_service.cc:954] ReplaceTablet: received request to replace tablet 1ba63c153ef04db891cae46a71439455 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:37.467458 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59508ee6bc6b47959d09f8f5aab89804" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:37.470788 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 90a57bc58e854e469897a04b2d434d06 with delete_type TABLET_DATA_DELETED (replaced by tablet 3223417aa5594ef7942c6d90bdc8711c) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:37.471210 1125 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 8d2f196f44be4fbc8059901380b71271 (table default.Table_360decd489b74f4b8ee93b008e57c45b [id=8cff6f6a2ebc447c979a45c9bfa438dc]). Expected version 2 got 0
I20260501 13:57:37.471889 2206 raft_consensus.cc:493] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:37.472127 2207 leader_election.cc:290] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:37.473156 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b3ac1f61eaf461384ae6a956ab56e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.473686 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b292910bf0464cb88efe70b5b0b730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.329293 1755 tablet_service.cc:1558] Processing DeleteTablet for tablet a9672aab85c64d548a4f6583a1e9af20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:37.472322 2206 raft_consensus.cc:515] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.475741 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet 90a57bc58e854e469897a04b2d434d06 with delete_type TABLET_DATA_DELETED (replaced by tablet 3223417aa5594ef7942c6d90bdc8711c) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:37.476264 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet 67b0675398d04be08bd29638b4ba2083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:37.309563 1757 tablet_service.cc:1558] Processing DeleteTablet for tablet b98187c7843e42b09d55c851117c4ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:37.375315 1753 tablet_service.cc:1511] Processing CreateTablet for tablet 0946834fac5f48a6a9976c21a45e0044 (DEFAULT_TABLE table=default.Table_65c41fbebfea4a3c8bacf216d517e869 [id=909a6cae06f54824bd1f573a8dc8e4c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.483047 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0946834fac5f48a6a9976c21a45e0044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.450356 2093 ts_tablet_manager.cc:1916] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:37.281361 1752 tablet_service.cc:1511] Processing CreateTablet for tablet 5a7eecc7345949cc81ad73e2d0370c9c (DEFAULT_TABLE table=default.Table_0abda464a00b44f49271dbde4c90e526 [id=3d3c9bce9beb4ed5b2614726e6886aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.487674 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a7eecc7345949cc81ad73e2d0370c9c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:37.289145 1717 consensus_peers.cc:597] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:37.490166 1881 raft_consensus.cc:359] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.491354 1881 raft_consensus.cc:385] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.491595 1881 raft_consensus.cc:740] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.304152 1756 tablet_service.cc:1511] Processing CreateTablet for tablet b125d1a190f947a5b2ef942f04408c51 (DEFAULT_TABLE table=default.Table_db50865558a14081b6f0280d9f45ceea [id=d9b238f8c49e4842b7055439df41703d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.495193 1606 tablet_service.cc:1511] Processing CreateTablet for tablet ef622e34d46a4b19a59e916d22c2b18b (DEFAULT_TABLE table=default.Table_4121e57a3a954fcd87c520271360371c [id=498581a9af3c4ca6a97a769fa38c53c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.495304 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 4b3ac1f61eaf461384ae6a956ab56e62 (DEFAULT_TABLE table=default.Table_907a34a78d554ca8a54102bcf65f7ed7 [id=65f926c9bcbd48fe898fe450f021dae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.496093 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef622e34d46a4b19a59e916d22c2b18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.280705 1754 tablet_service.cc:1511] Processing CreateTablet for tablet fd5d1afb88d14ddeb09a1c4d28324a35 (DEFAULT_TABLE table=default.Table_200f07afb8cf4eafbace88f5cd3090d2 [id=5c600e77298a4757a2801c398370d5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.497072 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b3ac1f61eaf461384ae6a956ab56e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.499826 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd5d1afb88d14ddeb09a1c4d28324a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.500383 1882 tablet_bootstrap.cc:492] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:37.500761 1882 ts_tablet_manager.cc:1403] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.146s user 0.011s sys 0.003s
I20260501 13:57:37.502204 1881 consensus_queue.cc:260] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.502992 1882 raft_consensus.cc:359] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.508558 1882 raft_consensus.cc:385] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.508819 1882 raft_consensus.cc:740] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.510726 1882 consensus_queue.cc:260] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.511639 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_41c886e2fe374f75b43ad2a436f9eb2b" } new_table_name: "default.Table_3e410e6cd57e44db8ad5287f606630fd" modify_external_catalogs: true
I20260501 13:57:37.521756 1461 tablet_service.cc:1511] Processing CreateTablet for tablet ef622e34d46a4b19a59e916d22c2b18b (DEFAULT_TABLE table=default.Table_4121e57a3a954fcd87c520271360371c [id=498581a9af3c4ca6a97a769fa38c53c0]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:37.493125 1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 204124us
I20260501 13:57:37.522778 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef622e34d46a4b19a59e916d22c2b18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.524212 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59508ee6bc6b47959d09f8f5aab89804" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:37.529502 1882 ts_tablet_manager.cc:1434] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.029s user 0.006s sys 0.001s
I20260501 13:57:37.531632 1882 tablet_bootstrap.cc:492] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:37.531646 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b125d1a190f947a5b2ef942f04408c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.539052 2206 leader_election.cc:290] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:37.539687 1881 ts_tablet_manager.cc:1434] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.103s user 0.007s sys 0.000s
I20260501 13:57:37.550664 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 1c630c25ec5c46539936ec545ff4d292 (DEFAULT_TABLE table=default.Table_4915b393002a47e895bfa0e9cd82da89 [id=67dd2e70eed74686b327b62e2cfae1a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.551774 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c630c25ec5c46539936ec545ff4d292. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:37.475836 1719 leader_election.cc:343] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:37.552194 1719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 76410.3us
W20260501 13:57:37.552891 1717 leader_election.cc:343] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:37.554713 1717 leader_election.cc:304] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:37.556674 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "406e1af8b7d9488f88697206daa87c05" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:37.558243 2207 raft_consensus.cc:2749] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:37.559568 1881 tablet_bootstrap.cc:492] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
W20260501 13:57:37.560446 1717 leader_election.cc:343] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:37.568766 1463 tablet_service.cc:1511] Processing CreateTablet for tablet 1c630c25ec5c46539936ec545ff4d292 (DEFAULT_TABLE table=default.Table_4915b393002a47e895bfa0e9cd82da89 [id=67dd2e70eed74686b327b62e2cfae1a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.572697 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c630c25ec5c46539936ec545ff4d292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.573345 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "406e1af8b7d9488f88697206daa87c05" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:37.575541 1757 tablet_service.cc:1511] Processing CreateTablet for tablet a2b292910bf0464cb88efe70b5b0b730 (DEFAULT_TABLE table=default.Table_7127557407564f3b824f44c92babdae3 [id=aa427a761f614ba8b607fc3cc39281a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.576696 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b292910bf0464cb88efe70b5b0b730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.583508 1755 tablet_service.cc:1511] Processing CreateTablet for tablet ef622e34d46a4b19a59e916d22c2b18b (DEFAULT_TABLE table=default.Table_4121e57a3a954fcd87c520271360371c [id=498581a9af3c4ca6a97a769fa38c53c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.584604 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef622e34d46a4b19a59e916d22c2b18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.588363 2255 tablet.cc:1724] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260501 13:57:37.598299 1465 tablet_service.cc:1511] Processing CreateTablet for tablet daca20eeafb141f2bf8b4233e0c8ea3b (DEFAULT_TABLE table=default.Table_430d0393c555431e8d75edfd188a80d6 [id=9bb928e7ec304e699ff70bd51b9cdfaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.599314 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daca20eeafb141f2bf8b4233e0c8ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:37.600478 1719 consensus_peers.cc:597] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260501 13:57:37.601159 1719 leader_election.cc:343] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:37.601572 1719 leader_election.cc:304] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:37.602491 1882 tablet_bootstrap.cc:654] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.596539 2256 tablet.cc:1724] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260501 13:57:37.602383 2207 raft_consensus.cc:2749] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:37.609822 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 152) took 1373 ms. Trace:
I20260501 13:57:37.610242 1899 rpcz_store.cc:276] 0501 13:57:36.236149 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:36.236329 (+ 180us) service_pool.cc:224] Handling call
0501 13:57:37.387743 (+1151414us) ts_tablet_manager.cc:1151] Deleting tablet 69626ed095bd4df7859563c176dafffc
0501 13:57:37.387789 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:37.609652 (+221863us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1150836}
I20260501 13:57:37.611186 1606 tablet_service.cc:1511] Processing CreateTablet for tablet daca20eeafb141f2bf8b4233e0c8ea3b (DEFAULT_TABLE table=default.Table_430d0393c555431e8d75edfd188a80d6 [id=9bb928e7ec304e699ff70bd51b9cdfaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.611655 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 153) took 1344 ms. Trace:
I20260501 13:57:37.611975 1899 rpcz_store.cc:276] 0501 13:57:36.267371 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:36.267496 (+ 125us) service_pool.cc:224] Handling call
0501 13:57:37.611332 (+1343836us) ts_tablet_manager.cc:1151] Deleting tablet 2c8d5d04a19443cfaa47b733b1d6ada6
0501 13:57:37.611374 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:37.611607 (+ 233us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1343326}
I20260501 13:57:37.613052 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 156) took 1124 ms. Trace:
I20260501 13:57:37.617375 1931 raft_consensus.cc:493] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:37.617802 1931 raft_consensus.cc:515] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.619568 1931 leader_election.cc:290] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:37.620445 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e41b43329e054442bc8697cc28161954" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:37.621150 1629 raft_consensus.cc:2468] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
W20260501 13:57:37.621680 1418 leader_election.cc:343] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: e41b43329e054442bc8697cc28161954
I20260501 13:57:37.622234 1421 leader_election.cc:304] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:37.623143 1931 raft_consensus.cc:2804] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:37.623445 1931 raft_consensus.cc:493] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:37.623741 1931 raft_consensus.cc:3060] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:37.624622 1882 tablet_bootstrap.cc:492] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:37.630105 1883 tablet_bootstrap.cc:492] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:37.630560 1883 ts_tablet_manager.cc:1403] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.463s user 0.039s sys 0.042s
I20260501 13:57:37.630618 1882 ts_tablet_manager.cc:1403] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.099s user 0.008s sys 0.001s
I20260501 13:57:37.637220 1124 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_16f2a19c6fb94677ad93beb76d5d9419" } modify_external_catalogs: true
I20260501 13:57:37.637742 1124 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_16f2a19c6fb94677ad93beb76d5d9419" } modify_external_catalogs: true
I20260501 13:57:37.643348 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daca20eeafb141f2bf8b4233e0c8ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.644819 1132 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 5e86d6c935d6444b8a953e7be1bdfa51 (table default.Table_ad4c78cb126f4013968e7b0e20a5c47c [id=0f9c1230865848adaa4e5ea744c31c8b]). Expected version 4 got 0
I20260501 13:57:37.645401 1132 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 406e1af8b7d9488f88697206daa87c05 (table default.Table_759f3009542147de94c73784e1698246 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]). Expected version 1 got 0
I20260501 13:57:37.645908 1132 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 8d2f196f44be4fbc8059901380b71271 (table default.Table_360decd489b74f4b8ee93b008e57c45b [id=8cff6f6a2ebc447c979a45c9bfa438dc]). Expected version 2 got 0
I20260501 13:57:37.643152 1931 raft_consensus.cc:515] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:37.647539 1719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 46430.7us
W20260501 13:57:37.648936 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 69626ed095bd4df7859563c176dafffc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69626ed095bd4df7859563c176dafffc already in progress: creating tablet
I20260501 13:57:37.644845 1883 raft_consensus.cc:359] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:37.649580 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 2c8d5d04a19443cfaa47b733b1d6ada6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c8d5d04a19443cfaa47b733b1d6ada6 already in progress: creating tablet
I20260501 13:57:37.649559 1757 tablet_service.cc:1511] Processing CreateTablet for tablet 1c630c25ec5c46539936ec545ff4d292 (DEFAULT_TABLE table=default.Table_4915b393002a47e895bfa0e9cd82da89 [id=67dd2e70eed74686b327b62e2cfae1a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.649958 1883 raft_consensus.cc:385] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.650305 1883 raft_consensus.cc:740] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.650627 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c630c25ec5c46539936ec545ff4d292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.651247 1883 consensus_queue.cc:260] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.653992 1883 ts_tablet_manager.cc:1434] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260501 13:57:37.656301 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e41b43329e054442bc8697cc28161954" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:37.656875 1629 raft_consensus.cc:3060] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:37.664458 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 1042bdf58c1f488daeaec284eb3319d5 (DEFAULT_TABLE table=default.Table_0c2d991875174eefbc2fa69108cc7082 [id=63f1fe234b1143869f21b76118b58234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.668757 1603 tablet_service.cc:1511] Processing CreateTablet for tablet 1042bdf58c1f488daeaec284eb3319d5 (DEFAULT_TABLE table=default.Table_0c2d991875174eefbc2fa69108cc7082 [id=63f1fe234b1143869f21b76118b58234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.669888 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1042bdf58c1f488daeaec284eb3319d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.681895 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1042bdf58c1f488daeaec284eb3319d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.683526 2047 ts_tablet_manager.cc:1929] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:37.683889 1881 tablet_bootstrap.cc:654] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.683985 2047 log.cc:1199] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/670f40234b8f43d485027818ac9694c8
I20260501 13:57:37.613348 1899 rpcz_store.cc:276] 0501 13:57:36.488956 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:36.489086 (+ 130us) service_pool.cc:224] Handling call
0501 13:57:37.612735 (+1123649us) ts_tablet_manager.cc:1151] Deleting tablet d361a2d82ecd456fb43f5e7fe3c83094
0501 13:57:37.612773 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:37.613002 (+ 229us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123042}
I20260501 13:57:37.685899 1899 tablet_replica.cc:333] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:37.687039 1899 raft_consensus.cc:2243] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:37.688021 1899 raft_consensus.cc:2272] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:37.720969 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet d361a2d82ecd456fb43f5e7fe3c83094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d361a2d82ecd456fb43f5e7fe3c83094 already in progress: creating tablet
W20260501 13:57:37.721589 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet d5d4064154c744198fa438cd3e3b7288 because the tablet was not found. No further retry: Not found: Tablet not found: d5d4064154c744198fa438cd3e3b7288
I20260501 13:57:37.721973 1132 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 6c2d5336bfec43dbb38ce7071834bca3 (table default.Table_66545ca4c3d540ada57273201e17f2d8 [id=697eaa78d5e541988cd48466c64c5915]). Expected version 1 got 0
W20260501 13:57:37.722117 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet e0356b19f0aa4197a41cc66d4440bffb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0356b19f0aa4197a41cc66d4440bffb already in progress: creating tablet
I20260501 13:57:37.727012 1624 raft_consensus.cc:3060] T f7645593dc184dab9f677f9371a88354 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:37.694386 1418 leader_election.cc:343] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: e41b43329e054442bc8697cc28161954
I20260501 13:57:37.721133 2207 raft_consensus.cc:493] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:37.654927 1883 tablet_bootstrap.cc:492] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:37.738667 1418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 44326.4us
I20260501 13:57:37.738935 2207 raft_consensus.cc:515] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.740859 2207 leader_election.cc:290] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:37.704516 1931 leader_election.cc:290] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:37.714047 2218 raft_consensus.cc:493] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:37.741623 2218 raft_consensus.cc:515] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.742234 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96e937b3a0b5451e8eaf4d5c47c17620" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:37.743495 1717 leader_election.cc:343] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:37.743631 2218 leader_election.cc:290] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:37.745695 1899 ts_tablet_manager.cc:1916] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:37.747635 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96e937b3a0b5451e8eaf4d5c47c17620" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:37.748929 2093 ts_tablet_manager.cc:1929] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:37.749257 2093 log.cc:1199] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/670f40234b8f43d485027818ac9694c8
I20260501 13:57:37.749926 2093 ts_tablet_manager.cc:1950] T 670f40234b8f43d485027818ac9694c8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:37.752606 1883 tablet_bootstrap.cc:654] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.752797 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 670f40234b8f43d485027818ac9694c8 (table default.Table_76959fdf8538441aa44f9e120288cdab [id=7803add44edc41d98147ce2f7ba60ab4]) successfully deleted
W20260501 13:57:37.753197 1421 leader_election.cc:343] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Not found: Tablet not found: adc47b3b2b084d33a12bd28cfad27382
W20260501 13:57:37.753959 1418 leader_election.cc:343] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: adc47b3b2b084d33a12bd28cfad27382
I20260501 13:57:37.754372 1418 leader_election.cc:304] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:37.726006 1882 raft_consensus.cc:359] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.755823 1882 raft_consensus.cc:385] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.756084 1882 raft_consensus.cc:740] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.756655 1882 consensus_queue.cc:260] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.765424 1881 tablet_bootstrap.cc:492] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:37.765841 1881 ts_tablet_manager.cc:1403] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.207s user 0.003s sys 0.006s
I20260501 13:57:37.766808 1629 raft_consensus.cc:2468] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:37.767220 1883 tablet_bootstrap.cc:492] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:37.767593 1883 ts_tablet_manager.cc:1403] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.113s user 0.005s sys 0.007s
I20260501 13:57:37.768340 1421 leader_election.cc:304] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:37.769369 2218 raft_consensus.cc:2749] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:37.770033 2218 raft_consensus.cc:2804] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:37.770503 2218 raft_consensus.cc:697] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:37.768371 1881 raft_consensus.cc:359] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.771320 1128 master_service.cc:954] ReplaceTablet: received request to replace tablet 19c1e0c30dce488baed9e046d90f0ba4 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:37.771611 1881 raft_consensus.cc:385] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.771783 2218 consensus_queue.cc:237] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.771860 1881 raft_consensus.cc:740] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.785048 2255 tablet.cc:1719] T 5e86d6c935d6444b8a953e7be1bdfa51 P 993b9ba18d8547f8af43167e727a6a24: Skipping requested alter to schema version 4, tablet already version 4
I20260501 13:57:37.758417 1882 ts_tablet_manager.cc:1434] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.127s user 0.005s sys 0.000s
I20260501 13:57:37.786697 1881 consensus_queue.cc:260] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.788686 1610 tablet_service.cc:1511] Processing CreateTablet for tablet 1bb7fdcad3a54317b43bfd58f08e89d0 (DEFAULT_TABLE table=default.Table_22d1c2cee41e4e9dad9e52bde00f1524 [id=a978daa811864093b075583f4e6bc4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.789677 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb7fdcad3a54317b43bfd58f08e89d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.791462 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 1bb7fdcad3a54317b43bfd58f08e89d0 (DEFAULT_TABLE table=default.Table_22d1c2cee41e4e9dad9e52bde00f1524 [id=a978daa811864093b075583f4e6bc4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.792459 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb7fdcad3a54317b43bfd58f08e89d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.792783 1882 tablet_bootstrap.cc:492] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:37.797672 1882 tablet_bootstrap.cc:654] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.799326 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 3b0ed592fd3642c194bb946c8c6003b4 (DEFAULT_TABLE table=default.Table_6e97f9768037452593b16b51f09a18ff [id=a5184058a20f431280f6ccd10a290f27]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:37.770684 1719 leader_election.cc:343] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:37.769933 1883 raft_consensus.cc:359] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.800099 1883 raft_consensus.cc:385] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.800310 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b0ed592fd3642c194bb946c8c6003b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.800395 1883 raft_consensus.cc:740] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.801007 1883 consensus_queue.cc:260] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.802942 1883 ts_tablet_manager.cc:1434] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.035s user 0.005s sys 0.002s
I20260501 13:57:37.803195 1881 ts_tablet_manager.cc:1434] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.034s user 0.004s sys 0.004s
I20260501 13:57:37.804006 1883 tablet_bootstrap.cc:492] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:37.806048 1881 tablet_bootstrap.cc:492] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:37.800117 1719 leader_election.cc:304] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:37.806666 2207 raft_consensus.cc:2749] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:37.809147 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 1042bdf58c1f488daeaec284eb3319d5 (DEFAULT_TABLE table=default.Table_0c2d991875174eefbc2fa69108cc7082 [id=63f1fe234b1143869f21b76118b58234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.810155 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1042bdf58c1f488daeaec284eb3319d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.811812 1883 tablet_bootstrap.cc:654] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.819840 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 3b0ed592fd3642c194bb946c8c6003b4 (DEFAULT_TABLE table=default.Table_6e97f9768037452593b16b51f09a18ff [id=a5184058a20f431280f6ccd10a290f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.820962 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b0ed592fd3642c194bb946c8c6003b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.774075 1755 tablet_service.cc:1511] Processing CreateTablet for tablet daca20eeafb141f2bf8b4233e0c8ea3b (DEFAULT_TABLE table=default.Table_430d0393c555431e8d75edfd188a80d6 [id=9bb928e7ec304e699ff70bd51b9cdfaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.790442 2256 tablet.cc:1719] T 5e86d6c935d6444b8a953e7be1bdfa51 P 0ed1f9a545474e76a661c32bf11a7fb2: Skipping requested alter to schema version 4, tablet already version 4
I20260501 13:57:37.825389 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daca20eeafb141f2bf8b4233e0c8ea3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.833906 1130 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 1042bdf58c1f488daeaec284eb3319d5
I20260501 13:57:37.841387 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 1042bdf58c1f488daeaec284eb3319d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:37 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:37.842483 1883 tablet_bootstrap.cc:492] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:37.842868 1883 ts_tablet_manager.cc:1403] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.002s
I20260501 13:57:37.842315 1603 tablet_service.cc:1558] Processing DeleteTablet for tablet 1042bdf58c1f488daeaec284eb3319d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:37 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:37.846859 1463 tablet_service.cc:1558] Processing DeleteTablet for tablet 1042bdf58c1f488daeaec284eb3319d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:37 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:37.846619 1883 raft_consensus.cc:359] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.850065 1883 raft_consensus.cc:385] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.850317 1883 raft_consensus.cc:740] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.851893 1881 tablet_bootstrap.cc:654] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.860538 2255 tablet.cc:1724] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:37.861296 1883 consensus_queue.cc:260] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.815387 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 3b0ed592fd3642c194bb946c8c6003b4 (DEFAULT_TABLE table=default.Table_6e97f9768037452593b16b51f09a18ff [id=a5184058a20f431280f6ccd10a290f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.863010 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b0ed592fd3642c194bb946c8c6003b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.864056 1883 ts_tablet_manager.cc:1434] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260501 13:57:37.864487 2256 tablet.cc:1724] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:37.865147 1883 tablet_bootstrap.cc:492] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:37.872943 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_2563926cdaef4f91a592f1261851a6b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:37.874797 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2563926cdaef4f91a592f1261851a6b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:37.738665 2047 ts_tablet_manager.cc:1950] T 670f40234b8f43d485027818ac9694c8 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:37.886256 2047 tablet_replica.cc:333] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:37.886773 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 670f40234b8f43d485027818ac9694c8 (table default.Table_76959fdf8538441aa44f9e120288cdab [id=7803add44edc41d98147ce2f7ba60ab4]) successfully deleted
I20260501 13:57:37.887692 1883 tablet_bootstrap.cc:654] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.900668 1755 tablet_service.cc:1511] Processing CreateTablet for tablet 1bb7fdcad3a54317b43bfd58f08e89d0 (DEFAULT_TABLE table=default.Table_22d1c2cee41e4e9dad9e52bde00f1524 [id=a978daa811864093b075583f4e6bc4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.902083 1603 tablet_service.cc:1511] Processing CreateTablet for tablet f57755ff94ce4e4481251c988eb337e5 (DEFAULT_TABLE table=default.Table_70769641e9fe4b9f9af063e534d0516c [id=01155632d70346beb9767446203bdf4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.902319 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb7fdcad3a54317b43bfd58f08e89d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.903172 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f57755ff94ce4e4481251c988eb337e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.903704 1882 tablet_bootstrap.cc:492] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:37.904121 1882 ts_tablet_manager.cc:1403] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.000s
I20260501 13:57:37.905684 2243 tablet.cc:1724] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:37.908882 1882 raft_consensus.cc:359] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.909507 1882 raft_consensus.cc:385] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.909746 1882 raft_consensus.cc:740] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.910326 1882 consensus_queue.cc:260] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.928032 1882 ts_tablet_manager.cc:1434] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260501 13:57:37.931916 1882 tablet_bootstrap.cc:492] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:37.934239 2093 tablet_replica.cc:333] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:37.935547 2093 raft_consensus.cc:2243] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:37.936064 2093 raft_consensus.cc:2272] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:37.936702 1883 tablet_bootstrap.cc:492] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:37.937129 1883 ts_tablet_manager.cc:1403] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.011s
I20260501 13:57:37.937906 2093 ts_tablet_manager.cc:1916] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:37.939417 1883 raft_consensus.cc:359] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.940024 1883 raft_consensus.cc:385] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:37.940274 1883 raft_consensus.cc:740] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:37.940840 1883 consensus_queue.cc:260] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:37.955312 1117 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ffe89d656956440e8d0c1de0b01ce486 consensus state has no leader
I20260501 13:57:37.956321 1883 ts_tablet_manager.cc:1434] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
W20260501 13:57:37.960271 1418 consensus_peers.cc:597] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 -> Peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Couldn't send request to peer 0ed1f9a545474e76a661c32bf11a7fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a563bccc7b046658ed133109f676528. This is attempt 1: this message will repeat every 5th retry.
W20260501 13:57:37.961284 1421 consensus_peers.cc:597] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:37.961337 2047 raft_consensus.cc:2243] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:37.962308 2047 pending_rounds.cc:70] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: Trying to abort 1 pending ops.
I20260501 13:57:37.962625 2047 pending_rounds.cc:77] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229238391382016 op_type: NO_OP noop_request { }
I20260501 13:57:37.962999 2047 raft_consensus.cc:2889] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:37.963555 2047 raft_consensus.cc:2272] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:37.964289 2207 raft_consensus.cc:493] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:37.965054 1883 tablet_bootstrap.cc:492] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:37.964721 2207 raft_consensus.cc:515] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:37.966971 2207 leader_election.cc:290] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:37.967322 1456 tablet_service.cc:1511] Processing CreateTablet for tablet f57755ff94ce4e4481251c988eb337e5 (DEFAULT_TABLE table=default.Table_70769641e9fe4b9f9af063e534d0516c [id=01155632d70346beb9767446203bdf4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:37.968305 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f57755ff94ce4e4481251c988eb337e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:37.970077 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec4bd4cb1e70472aa09fc93afc3b1501" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:37.973575 1717 leader_election.cc:343] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:37.991918 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_3339262cb6b34f56a02f22d29bdd2b14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:37.993424 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3339262cb6b34f56a02f22d29bdd2b14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:38.000345 2207 raft_consensus.cc:493] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:38.000789 2207 raft_consensus.cc:515] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.002714 2207 leader_election.cc:290] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:38.004148 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0669a229d6d54eacac9889ad18e06f30" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:38.005604 1717 leader_election.cc:343] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.011713 1756 tablet_service.cc:1511] Processing CreateTablet for tablet f57755ff94ce4e4481251c988eb337e5 (DEFAULT_TABLE table=default.Table_70769641e9fe4b9f9af063e534d0516c [id=01155632d70346beb9767446203bdf4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:38.015023 1899 ts_tablet_manager.cc:1929] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:38.015494 1899 log.cc:1199] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/670f40234b8f43d485027818ac9694c8
I20260501 13:57:38.016157 1899 ts_tablet_manager.cc:1950] T 670f40234b8f43d485027818ac9694c8 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:37.944094 1882 tablet_bootstrap.cc:654] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:37.968217 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec4bd4cb1e70472aa09fc93afc3b1501" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:38.003312 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0669a229d6d54eacac9889ad18e06f30" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:38.021351 1719 leader_election.cc:343] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.021739 1719 leader_election.cc:304] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:38.022672 1719 leader_election.cc:343] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.022987 1719 leader_election.cc:304] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:38.024012 2047 ts_tablet_manager.cc:1916] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:38.023955 2206 raft_consensus.cc:2749] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:38.024322 2207 raft_consensus.cc:2749] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:38.035964 1883 tablet_bootstrap.cc:654] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:38.039727 1882 tablet_bootstrap.cc:492] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:38.040227 1882 ts_tablet_manager.cc:1403] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.109s user 0.012s sys 0.005s
I20260501 13:57:38.041661 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f57755ff94ce4e4481251c988eb337e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:38.042675 1882 raft_consensus.cc:359] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.043435 1882 raft_consensus.cc:385] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.043728 1882 raft_consensus.cc:740] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.044411 1882 consensus_queue.cc:260] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.058962 1882 ts_tablet_manager.cc:1434] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260501 13:57:38.062430 1882 tablet_bootstrap.cc:492] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:38.068436 1882 tablet_bootstrap.cc:654] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:38.071535 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c630c25ec5c46539936ec545ff4d292 consensus state has no leader
I20260501 13:57:38.072486 1129 master_service.cc:954] ReplaceTablet: received request to replace tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:38.101143 1882 tablet_bootstrap.cc:492] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:38.101548 1882 ts_tablet_manager.cc:1403] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.012s
I20260501 13:57:38.103890 1882 raft_consensus.cc:359] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.104494 1882 raft_consensus.cc:385] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.104732 1882 raft_consensus.cc:740] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.105283 1882 consensus_queue.cc:260] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.113262 1881 tablet_bootstrap.cc:492] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:38.113917 1881 ts_tablet_manager.cc:1403] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.308s user 0.007s sys 0.006s
I20260501 13:57:38.117954 1882 ts_tablet_manager.cc:1434] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260501 13:57:38.121079 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 670f40234b8f43d485027818ac9694c8 (table default.Table_76959fdf8538441aa44f9e120288cdab [id=7803add44edc41d98147ce2f7ba60ab4]) successfully deleted
I20260501 13:57:38.142663 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_2a1ca2e987e243ad9f1c064a28866dd3" } modify_external_catalogs: true
I20260501 13:57:38.143234 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_2a1ca2e987e243ad9f1c064a28866dd3" } modify_external_catalogs: true
I20260501 13:57:38.159848 2214 raft_consensus.cc:493] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:38.160332 2214 raft_consensus.cc:515] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.164698 1127 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_66545ca4c3d540ada57273201e17f2d8 [id=697eaa78d5e541988cd48466c64c5915] alter complete (version 1)
I20260501 13:57:38.162220 2214 leader_election.cc:290] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:38.166067 2214 raft_consensus.cc:493] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:38.166438 2214 raft_consensus.cc:515] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.168766 2214 leader_election.cc:290] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:38.170131 1421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 53943.5us
I20260501 13:57:38.173559 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ff05404061447199d936d005afda653" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:38.174124 1482 raft_consensus.cc:2468] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:38.174988 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c33a1f423bdb44f5843daffe8f73e74a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:38.177381 1564 leader_election.cc:304] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
W20260501 13:57:38.178212 1564 leader_election.cc:343] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.180438 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c33a1f423bdb44f5843daffe8f73e74a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:38.181104 1782 raft_consensus.cc:2468] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
W20260501 13:57:38.182984 1563 leader_election.cc:343] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 2ff05404061447199d936d005afda653
I20260501 13:57:38.183866 1563 leader_election.cc:304] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:38.185019 2214 raft_consensus.cc:2804] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:38.185395 2214 raft_consensus.cc:493] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:38.185725 2214 raft_consensus.cc:3060] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:38.194236 2214 raft_consensus.cc:515] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:38.195776 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 because the tablet was not found. No further retry: Not found: Tablet not found: 4f2448ee6bf14bbc8ec0e1d59ccb7b90
I20260501 13:57:38.196532 1899 tablet_replica.cc:333] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:38.197451 1899 raft_consensus.cc:2243] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:38.198004 1899 raft_consensus.cc:2272] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:38.199822 1899 ts_tablet_manager.cc:1916] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:38.207958 1881 raft_consensus.cc:359] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.208818 1881 raft_consensus.cc:385] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.209148 1881 raft_consensus.cc:740] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.209918 1881 consensus_queue.cc:260] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.201552 1883 tablet_bootstrap.cc:492] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:38.231834 2207 raft_consensus.cc:493] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:38.202648 2273 raft_consensus.cc:2804] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:38.238819 2207 raft_consensus.cc:515] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.239199 2273 raft_consensus.cc:493] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:38.239674 2273 raft_consensus.cc:3060] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:38.240643 2207 leader_election.cc:290] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:38.211126 1882 tablet_bootstrap.cc:492] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:38.238867 1883 ts_tablet_manager.cc:1403] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.274s user 0.011s sys 0.001s
I20260501 13:57:38.245139 1883 raft_consensus.cc:359] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.231361 1629 raft_consensus.cc:3060] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:38.246286 1883 raft_consensus.cc:385] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.246724 1883 raft_consensus.cc:740] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.248541 2214 leader_election.cc:290] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:38.250279 1563 leader_election.cc:343] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 2ff05404061447199d936d005afda653
I20260501 13:57:38.251263 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33a355897309420c97b1257afaefe8d1" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:38.254940 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ff05404061447199d936d005afda653" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:38.255470 1483 raft_consensus.cc:3060] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:38.256579 1881 ts_tablet_manager.cc:1434] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.142s user 0.006s sys 0.000s
I20260501 13:57:38.257023 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33a355897309420c97b1257afaefe8d1" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:38.257437 1881 tablet_bootstrap.cc:492] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
W20260501 13:57:38.261409 1127 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.129:53385 (request call id 54) took 841 ms (client timeout 1000 ms). Trace:
W20260501 13:57:38.268582 1125 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.130:35289 (request call id 60) took 802 ms (client timeout 1000 ms). Trace:
W20260501 13:57:38.268921 1125 rpcz_store.cc:269] 0501 13:57:37.465783 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.465908 (+ 125us) service_pool.cc:224] Handling call
0501 13:57:38.268523 (+802615us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":5}
I20260501 13:57:38.271458 1882 tablet_bootstrap.cc:654] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:38.272506 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet adc47b3b2b084d33a12bd28cfad27382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:38.275522 1132 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.131:42895 (request call id 58) took 931 ms (client timeout 1000 ms). Trace:
W20260501 13:57:38.275832 1132 rpcz_store.cc:269] 0501 13:57:37.343816 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.346568 (+ 2752us) service_pool.cc:224] Handling call
0501 13:57:38.275469 (+928901us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8471,"reported_tablets":6}
I20260501 13:57:38.276108 1483 raft_consensus.cc:2468] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:38.277220 1564 leader_election.cc:304] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
W20260501 13:57:38.280333 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9018d70c734a429aa59565ad67a46acc consensus state has no leader
I20260501 13:57:38.281261 1124 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet a9caca85ce714d0cab25ae1a35fd618f
I20260501 13:57:38.288576 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_94e9163950b1491b8dbba021591c6123" } new_table_name: "default.Table_ce141ed4961d49a8a07433b1dded9ce5" modify_external_catalogs: true
I20260501 13:57:38.290151 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet e41b43329e054442bc8697cc28161954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:38.290607 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_43f068125a9243dbadd21b551da3226b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:38.292011 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43f068125a9243dbadd21b551da3226b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:38.235100 2274 raft_consensus.cc:493] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:38.270032 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_ef8d1ad25b3a469cb3d3840b998178d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:38.293447 2274 raft_consensus.cc:515] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.295789 2274 leader_election.cc:290] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:38.296926 1563 leader_election.cc:343] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 104708eb7fc0412dad0bd7e14eee9d13
W20260501 13:57:38.261787 1127 rpcz_store.cc:269] 0501 13:57:37.420228 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.420460 (+ 232us) service_pool.cc:224] Handling call
0501 13:57:38.261333 (+840873us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:38.084442 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:38.084787 (+ 345us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:38.084809 (+ 22us) write_op.cc:435] Acquired schema lock
0501 13:57:38.084821 (+ 12us) tablet.cc:662] Decoding operations
0501 13:57:38.085092 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:38.085128 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:38.085150 (+ 22us) tablet.cc:685] Acquiring locks for 1 operations
0501 13:57:38.085280 (+ 130us) tablet.cc:701] Row locks acquired
0501 13:57:38.085293 (+ 13us) write_op.cc:260] PREPARE: finished
0501 13:57:38.085384 (+ 91us) write_op.cc:270] Start()
0501 13:57:38.085462 (+ 78us) write_op.cc:276] Timestamp: P: 1777643858085387 usec, L: 0
0501 13:57:38.085478 (+ 16us) op_driver.cc:348] REPLICATION: starting
0501 13:57:38.140047 (+ 54569us) log.cc:844] Serialized 356 byte log entry
0501 13:57:38.151559 (+ 11512us) spinlock_profiling.cc:243] Waited 54 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663024951 00007f56630b7208 00007f56635299f1 00007f56635291c8 00007f566352de61 00007f566352de19 00007f566352dda9 00007f566352dd71 00007f566352cf9d 00000000005b3bc4 00000000005b39f9
0501 13:57:38.160152 (+ 8593us) op_driver.cc:464] REPLICATION: finished
0501 13:57:38.162712 (+ 2560us) write_op.cc:301] APPLY: starting
0501 13:57:38.162766 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:38.162845 (+ 79us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:38.162863 (+ 18us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:38.163040 (+ 177us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:38.163164 (+ 124us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:57:38.163204 (+ 40us) write_op.cc:312] APPLY: finished
0501 13:57:38.163352 (+ 148us) log.cc:844] Serialized 23 byte log entry
0501 13:57:38.163489 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:38.163546 (+ 57us) write_op.cc:454] Released schema lock
0501 13:57:38.163574 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:38.165574 (+ 2000us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:38.165598 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":2388,"apply.run_cpu_time_us":1069,"apply.run_wall_time_us":1068,"num_ops":1,"prepare.queue_time_us":646444,"prepare.run_cpu_time_us":3585,"prepare.run_wall_time_us":81123,"raft.queue_time_us":115,"raft.run_cpu_time_us":708,"raft.run_wall_time_us":13149,"replication_time_us":74584,"spinlock_wait_cycles":124313472}]]}
W20260501 13:57:38.298476 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef8d1ad25b3a469cb3d3840b998178d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:38.296279 2274 raft_consensus.cc:493] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:38.311944 2274 raft_consensus.cc:515] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:38.315696 1563 leader_election.cc:343] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 2ee580bb9d3843f9bbeae8eef2c347ad
I20260501 13:57:38.316272 2274 leader_election.cc:290] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:38.316785 2274 raft_consensus.cc:2804] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:38.317224 2274 raft_consensus.cc:697] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:38.333916 1128 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 19c1e0c30dce488baed9e046d90f0ba4
I20260501 13:57:38.336212 1128 catalog_manager.cc:6741] ReplaceTablet: tablet 19c1e0c30dce488baed9e046d90f0ba4 deleted and replaced by tablet ef191926320a4e0fa8bd94e47e9b9a86
I20260501 13:57:38.351475 1456 tablet_service.cc:1558] Processing DeleteTablet for tablet 19c1e0c30dce488baed9e046d90f0ba4 with delete_type TABLET_DATA_DELETED (replaced by tablet ef191926320a4e0fa8bd94e47e9b9a86) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:38.258509 1717 leader_election.cc:343] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:38.353046 1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 94606.5us
I20260501 13:57:38.262738 2207 raft_consensus.cc:493] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 13:57:38.354094 1719 leader_election.cc:343] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.355710 1719 leader_election.cc:304] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:38.356200 1883 consensus_queue.cc:260] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.353631 2207 raft_consensus.cc:515] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:38.358341 1719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 99071.6us
I20260501 13:57:38.358778 2206 raft_consensus.cc:2749] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:38.371848 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet 19c1e0c30dce488baed9e046d90f0ba4 with delete_type TABLET_DATA_DELETED (replaced by tablet ef191926320a4e0fa8bd94e47e9b9a86) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:38.371991 2207 leader_election.cc:290] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:38.373548 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c789ca9e3c748d5bc79f4e1b0ea8fd1" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:38.374730 1717 leader_election.cc:343] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.374766 2206 raft_consensus.cc:493] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:38.375557 2206 raft_consensus.cc:515] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.379384 2206 leader_election.cc:290] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:38.381085 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9abb39c8b44d4b4c8b69751a4cdd6668" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:38.382970 1717 leader_election.cc:343] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.411470 1883 ts_tablet_manager.cc:1434] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.169s user 0.004s sys 0.004s
I20260501 13:57:38.412110 2047 ts_tablet_manager.cc:1929] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:38.412410 1883 tablet_bootstrap.cc:492] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:38.412564 2047 log.cc:1199] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/4a563bccc7b046658ed133109f676528
I20260501 13:57:38.413386 2047 ts_tablet_manager.cc:1950] T 4a563bccc7b046658ed133109f676528 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:38.320494 2273 raft_consensus.cc:515] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:38.343901 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 66762.8us
I20260501 13:57:38.419008 2273 leader_election.cc:290] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:38.419828 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c33a1f423bdb44f5843daffe8f73e74a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:38.420387 1782 raft_consensus.cc:3060] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:38.423177 1883 tablet_bootstrap.cc:654] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:38.423734 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee580bb9d3843f9bbeae8eef2c347ad" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:38.428237 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c33a1f423bdb44f5843daffe8f73e74a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:38.428758 1484 raft_consensus.cc:3060] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:38.432089 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "104708eb7fc0412dad0bd7e14eee9d13" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:38.433082 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c789ca9e3c748d5bc79f4e1b0ea8fd1" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:38.434765 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9abb39c8b44d4b4c8b69751a4cdd6668" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:38.436007 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet 19c1e0c30dce488baed9e046d90f0ba4 with delete_type TABLET_DATA_DELETED (replaced by tablet ef191926320a4e0fa8bd94e47e9b9a86) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:38.436314 1881 tablet_bootstrap.cc:654] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:38.436899 1719 leader_election.cc:343] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.437297 1719 leader_election.cc:304] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:38.437903 2282 raft_consensus.cc:2749] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:38.438524 1719 leader_election.cc:343] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.438876 1719 leader_election.cc:304] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:38.439374 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 159) took 1032 ms. Trace:
I20260501 13:57:38.439440 2282 raft_consensus.cc:2749] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:38.439731 2047 rpcz_store.cc:276] 0501 13:57:37.406777 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.409807 (+ 3030us) service_pool.cc:224] Handling call
0501 13:57:37.886111 (+476304us) ts_tablet_manager.cc:1151] Deleting tablet 4a563bccc7b046658ed133109f676528
0501 13:57:37.886151 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:38.219347 (+333196us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:38.411975 (+192628us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:38.439317 (+ 27342us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":475772}
I20260501 13:57:38.440963 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 4a563bccc7b046658ed133109f676528 (table default.Table_9e3d29db155f4fd2ba6229fc2318edb0 [id=08c0033dd652465797b8be473d0e427f]) successfully deleted
I20260501 13:57:38.441903 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 160) took 1029 ms. Trace:
I20260501 13:57:38.442322 2047 rpcz_store.cc:276] 0501 13:57:37.412782 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.414554 (+ 1772us) service_pool.cc:224] Handling call
0501 13:57:38.441589 (+1027035us) ts_tablet_manager.cc:1151] Deleting tablet 67b0675398d04be08bd29638b4ba2083
0501 13:57:38.441630 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:38.441854 (+ 224us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026499}
I20260501 13:57:38.443153 1127 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet c33a1f423bdb44f5843daffe8f73e74a (table default.Table_ecd46a159d2448f9ba09fc99e14a22a3 [id=d98d5437c5bc4e44bcfe27983dfa36e6]). Expected version 1 got 0
W20260501 13:57:38.445374 1564 leader_election.cc:343] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.445791 1564 leader_election.cc:304] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:38.446719 2273 raft_consensus.cc:2749] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:38.447594 1564 leader_election.cc:343] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.448117 1564 leader_election.cc:304] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:38.325148 2274 consensus_queue.cc:237] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.409907 2243 tablet.cc:1724] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260501 13:57:38.363241 1882 tablet_bootstrap.cc:492] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:38.459894 1882 ts_tablet_manager.cc:1403] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.249s user 0.010s sys 0.004s
I20260501 13:57:38.361811 2093 ts_tablet_manager.cc:1929] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:38.462580 1782 raft_consensus.cc:2468] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:38.462881 2093 log.cc:1199] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/4a563bccc7b046658ed133109f676528
I20260501 13:57:38.463775 1563 leader_election.cc:304] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:38.464273 2093 ts_tablet_manager.cc:1950] T 4a563bccc7b046658ed133109f676528 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:38.471701 2276 raft_consensus.cc:2749] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:38.471861 2273 raft_consensus.cc:2804] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:38.474152 1882 raft_consensus.cc:359] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.479498 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1
I20260501 13:57:38.479759 1882 raft_consensus.cc:385] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.480031 1882 raft_consensus.cc:740] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.480677 1882 consensus_queue.cc:260] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.483875 1129 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 94abe840b1994c0cb1f7ac7f4ac09cd9
I20260501 13:57:38.485100 1129 catalog_manager.cc:6741] ReplaceTablet: tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 deleted and replaced by tablet f28db04617a449af8d87a554fb63e1e4
I20260501 13:57:38.492456 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:38.493640 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 with delete_type TABLET_DATA_DELETED (replaced by tablet f28db04617a449af8d87a554fb63e1e4) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:38.497022 1882 ts_tablet_manager.cc:1434] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20260501 13:57:38.498859 1882 tablet_bootstrap.cc:492] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:38.501145 2282 raft_consensus.cc:493] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:38.501585 2282 raft_consensus.cc:515] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.502951 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 171) took 1091 ms. Trace:
I20260501 13:57:38.503403 2282 leader_election.cc:290] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:38.503223 2093 rpcz_store.cc:276] 0501 13:57:37.411560 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.416444 (+ 4884us) service_pool.cc:224] Handling call
0501 13:57:37.934092 (+517648us) ts_tablet_manager.cc:1151] Deleting tablet 4a563bccc7b046658ed133109f676528
0501 13:57:37.934135 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:38.152742 (+218607us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:38.361661 (+208919us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:38.502886 (+141225us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":487023}
I20260501 13:57:38.507191 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 172) took 1088 ms. Trace:
I20260501 13:57:38.514190 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 4a563bccc7b046658ed133109f676528 (table default.Table_9e3d29db155f4fd2ba6229fc2318edb0 [id=08c0033dd652465797b8be473d0e427f]) successfully deleted
I20260501 13:57:38.524042 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:38.525539 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 with delete_type TABLET_DATA_DELETED (replaced by tablet f28db04617a449af8d87a554fb63e1e4) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:38.532145 2243 tablet.cc:1719] T 5e86d6c935d6444b8a953e7be1bdfa51 P 8f90b7ff170b4eadb27abc9ef866ffcf: Skipping requested alter to schema version 4, tablet already version 4
I20260501 13:57:38.535642 2273 raft_consensus.cc:697] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:38.536907 2273 consensus_queue.cc:237] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.545320 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eb6d30bed2a410db8b62fdff54a5fac" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:38.543247 2093 rpcz_store.cc:276] 0501 13:57:37.418158 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.418276 (+ 118us) service_pool.cc:224] Handling call
0501 13:57:38.506313 (+1088037us) ts_tablet_manager.cc:1151] Deleting tablet 67b0675398d04be08bd29638b4ba2083
0501 13:57:38.506383 (+ 70us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:38.506879 (+ 496us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1059020}
I20260501 13:57:38.444450 1484 raft_consensus.cc:2468] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:38.489282 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 with delete_type TABLET_DATA_DELETED (replaced by tablet f28db04617a449af8d87a554fb63e1e4) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:38.548250 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 67b0675398d04be08bd29638b4ba2083 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67b0675398d04be08bd29638b4ba2083 already in progress: creating tablet
I20260501 13:57:38.494117 1456 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:38.504469 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eb6d30bed2a410db8b62fdff54a5fac" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:38.549304 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 161) took 1118 ms. Trace:
W20260501 13:57:38.549593 1717 leader_election.cc:343] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.549600 2047 rpcz_store.cc:276] 0501 13:57:37.431101 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.475598 (+ 44497us) service_pool.cc:224] Handling call
0501 13:57:38.549017 (+1073419us) ts_tablet_manager.cc:1151] Deleting tablet 90a57bc58e854e469897a04b2d434d06
0501 13:57:38.549057 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:38.549253 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1072885}
W20260501 13:57:38.550755 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 90a57bc58e854e469897a04b2d434d06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90a57bc58e854e469897a04b2d434d06 already in progress: creating tablet
W20260501 13:57:38.552218 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 1042bdf58c1f488daeaec284eb3319d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1042bdf58c1f488daeaec284eb3319d5 already in progress: creating tablet
I20260501 13:57:38.553131 2047 tablet_replica.cc:333] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:38.553975 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 173) took 1115 ms. Trace:
W20260501 13:57:38.555183 1719 leader_election.cc:343] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.554276 2093 rpcz_store.cc:276] 0501 13:57:37.432632 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.432753 (+ 121us) service_pool.cc:224] Handling call
0501 13:57:38.547390 (+1114637us) ts_tablet_manager.cc:1151] Deleting tablet 90a57bc58e854e469897a04b2d434d06
0501 13:57:38.547428 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:38.547637 (+ 209us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1098415}
I20260501 13:57:38.555615 1719 leader_election.cc:304] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:38.561057 1899 ts_tablet_manager.cc:1929] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260501 13:57:38.562530 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 67b0675398d04be08bd29638b4ba2083 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67b0675398d04be08bd29638b4ba2083 already in progress: creating tablet
W20260501 13:57:38.564323 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 90a57bc58e854e469897a04b2d434d06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90a57bc58e854e469897a04b2d434d06 already in progress: creating tablet
I20260501 13:57:38.567581 1881 tablet_bootstrap.cc:492] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:38.568140 1881 ts_tablet_manager.cc:1403] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.311s user 0.008s sys 0.002s
I20260501 13:57:38.570454 1881 raft_consensus.cc:359] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.571038 2093 tablet_replica.cc:333] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
W20260501 13:57:38.574193 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 1042bdf58c1f488daeaec284eb3319d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1042bdf58c1f488daeaec284eb3319d5 already in progress: creating tablet
I20260501 13:57:38.574998 2093 raft_consensus.cc:2243] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:38.575460 2093 raft_consensus.cc:2272] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:38.577180 2093 ts_tablet_manager.cc:1916] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:38.578696 1882 tablet_bootstrap.cc:654] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:38.586421 1130 master_service.cc:954] ReplaceTablet: received request to replace tablet 0669a229d6d54eacac9889ad18e06f30 from {username='slave'} at 127.0.0.1:41700
W20260501 13:57:38.594782 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8efee17e2c34caf98c0ede01d2a943a consensus state has no leader
I20260501 13:57:38.605400 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6e97f9768037452593b16b51f09a18ff" } new_table_name: "default.Table_407a6cfd065142f4a2debee031077ab9" modify_external_catalogs: true
I20260501 13:57:38.607815 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_e7f9e5733a384b468efb94f393f29328"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:38.609565 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7f9e5733a384b468efb94f393f29328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:38.612042 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_5bb0715c4b1e4e78bbcd523962cf53d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:38.613749 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bb0715c4b1e4e78bbcd523962cf53d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:38.617449 1125 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_3339262cb6b34f56a02f22d29bdd2b14" } modify_external_catalogs: true
I20260501 13:57:38.618011 1125 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_3339262cb6b34f56a02f22d29bdd2b14" } modify_external_catalogs: true
I20260501 13:57:38.640278 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet c33a1f423bdb44f5843daffe8f73e74a (table default.Table_ecd46a159d2448f9ba09fc99e14a22a3 [id=d98d5437c5bc4e44bcfe27983dfa36e6]). Expected version 1 got 0
I20260501 13:57:38.651901 1882 tablet_bootstrap.cc:492] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:38.576359 2206 raft_consensus.cc:2749] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:38.652292 1882 ts_tablet_manager.cc:1403] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.154s user 0.005s sys 0.007s
I20260501 13:57:38.578953 1883 tablet_bootstrap.cc:492] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:38.652684 1899 log.cc:1199] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/90a57bc58e854e469897a04b2d434d06
I20260501 13:57:38.652815 1883 ts_tablet_manager.cc:1403] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.241s user 0.023s sys 0.019s
I20260501 13:57:38.654685 1882 raft_consensus.cc:359] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.655032 1899 ts_tablet_manager.cc:1950] T 90a57bc58e854e469897a04b2d434d06 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:38.655295 1882 raft_consensus.cc:385] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.655628 1882 raft_consensus.cc:740] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.656177 1882 consensus_queue.cc:260] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.638448 2282 raft_consensus.cc:493] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:38.660986 1882 ts_tablet_manager.cc:1434] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20260501 13:57:38.660830 2282 raft_consensus.cc:515] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.662047 1881 raft_consensus.cc:385] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.662645 2282 leader_election.cc:290] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:38.656725 1883 raft_consensus.cc:359] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.662957 1881 raft_consensus.cc:740] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.663208 1883 raft_consensus.cc:385] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.663451 1883 raft_consensus.cc:740] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.663645 1881 consensus_queue.cc:260] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.664013 1883 consensus_queue.cc:260] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.664336 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1409fbe5846a4bb099f31fb2d2e41382" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:38.649600 2047 raft_consensus.cc:2243] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:38.665253 2047 raft_consensus.cc:2272] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:38.665795 1883 ts_tablet_manager.cc:1434] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260501 13:57:38.668437 1883 tablet_bootstrap.cc:492] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:38.670823 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1409fbe5846a4bb099f31fb2d2e41382" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:38.672200 2047 ts_tablet_manager.cc:1916] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:38.672297 1717 leader_election.cc:343] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.677712 1881 ts_tablet_manager.cc:1434] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.109s user 0.005s sys 0.000s
W20260501 13:57:38.679852 1719 leader_election.cc:343] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.680227 1719 leader_election.cc:304] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:38.680867 2282 raft_consensus.cc:2749] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:38.679314 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 168) took 1209 ms. Trace:
I20260501 13:57:38.685168 1131 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_360decd489b74f4b8ee93b008e57c45b" } modify_external_catalogs: true
I20260501 13:57:38.685740 1131 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_360decd489b74f4b8ee93b008e57c45b" } modify_external_catalogs: true
I20260501 13:57:38.684088 1899 rpcz_store.cc:276] 0501 13:57:37.470378 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.470637 (+ 259us) service_pool.cc:224] Handling call
0501 13:57:38.195489 (+724852us) ts_tablet_manager.cc:1151] Deleting tablet 90a57bc58e854e469897a04b2d434d06
0501 13:57:38.196422 (+ 933us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:38.429522 (+233100us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:38.560917 (+131395us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:38.679257 (+118340us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":724241}
I20260501 13:57:38.687273 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 167) took 1221 ms. Trace:
I20260501 13:57:38.687577 1899 rpcz_store.cc:276] 0501 13:57:37.466348 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.466450 (+ 102us) service_pool.cc:224] Handling call
0501 13:57:38.686939 (+1220489us) ts_tablet_manager.cc:1151] Deleting tablet 67b0675398d04be08bd29638b4ba2083
0501 13:57:38.686983 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:38.687222 (+ 239us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210344}
I20260501 13:57:38.687786 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 90a57bc58e854e469897a04b2d434d06 (table default.Table_f8e9d88347b54e2da903776319f1c181 [id=b3e0852bf1bc4eedad8b69f7e4cf0785]) successfully deleted
I20260501 13:57:38.688432 1899 tablet_replica.cc:333] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
W20260501 13:57:38.688997 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 67b0675398d04be08bd29638b4ba2083 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67b0675398d04be08bd29638b4ba2083 already in progress: creating tablet
I20260501 13:57:38.692183 1883 tablet_bootstrap.cc:654] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:38.695029 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_a1116dca3a314f4b8508c62da88fbdc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:38.696574 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1116dca3a314f4b8508c62da88fbdc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:38.696624 1899 raft_consensus.cc:2243] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:38.697080 1899 raft_consensus.cc:2272] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:38.698711 1899 ts_tablet_manager.cc:1916] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:38.701763 1882 tablet_bootstrap.cc:492] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:38.711354 1881 tablet_bootstrap.cc:492] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:38.720170 1883 tablet_bootstrap.cc:492] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:38.720590 1883 ts_tablet_manager.cc:1403] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.004s
I20260501 13:57:38.722908 1883 raft_consensus.cc:359] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.723479 1883 raft_consensus.cc:385] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.723707 1883 raft_consensus.cc:740] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.724728 2093 ts_tablet_manager.cc:1929] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:38.725080 2093 log.cc:1199] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/e41b43329e054442bc8697cc28161954
I20260501 13:57:38.725663 2093 ts_tablet_manager.cc:1950] T e41b43329e054442bc8697cc28161954 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:38.734534 1883 consensus_queue.cc:260] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.746992 1881 tablet_bootstrap.cc:654] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:38.748593 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet e41b43329e054442bc8697cc28161954 (table default.Table_2edf812ced814de8acadfe87bcad5893 [id=694ff2769ea04445811ca8d8a92854fe]) successfully deleted
I20260501 13:57:38.749751 1883 ts_tablet_manager.cc:1434] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260501 13:57:38.755434 1883 tablet_bootstrap.cc:492] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:38.754407 1882 tablet_bootstrap.cc:654] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:38.757869 1418 consensus_peers.cc:597] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 -> Peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Couldn't send request to peer 0ed1f9a545474e76a661c32bf11a7fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e41b43329e054442bc8697cc28161954. This is attempt 1: this message will repeat every 5th retry.
W20260501 13:57:38.758500 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 19c1e0c30dce488baed9e046d90f0ba4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19c1e0c30dce488baed9e046d90f0ba4 already in progress: creating tablet
I20260501 13:57:38.759588 2282 raft_consensus.cc:493] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:38.760006 2282 raft_consensus.cc:515] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.761725 2282 leader_election.cc:290] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:38.762392 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "920d7f9748ae403d90573d612feeba59" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:38.762737 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 already in progress: creating tablet
I20260501 13:57:38.763003 1629 raft_consensus.cc:2468] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 0.
I20260501 13:57:38.763109 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "920d7f9748ae403d90573d612feeba59" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:38.764128 1719 leader_election.cc:304] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
W20260501 13:57:38.764659 1717 leader_election.cc:343] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.764845 2282 raft_consensus.cc:2804] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:38.765151 2282 raft_consensus.cc:493] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:38.765427 2282 raft_consensus.cc:3060] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:38.768965 2047 ts_tablet_manager.cc:1929] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:38.771968 2282 raft_consensus.cc:515] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.774658 2282 leader_election.cc:290] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:38.774564 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "920d7f9748ae403d90573d612feeba59" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:38.775076 1629 raft_consensus.cc:3060] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:38.775936 1125 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet cf4017555f1d486db20bbf4fb6198302
I20260501 13:57:38.776510 1125 master_service.cc:954] ReplaceTablet: received request to replace tablet ec4bd4cb1e70472aa09fc93afc3b1501 from {username='slave'} at 127.0.0.1:41700
W20260501 13:57:38.779663 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b0ed592fd3642c194bb946c8c6003b4 consensus state has no leader
I20260501 13:57:38.788506 1882 tablet_bootstrap.cc:492] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:38.788897 1882 ts_tablet_manager.cc:1403] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.001s
I20260501 13:57:38.792006 1899 ts_tablet_manager.cc:1929] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:38.792384 1899 log.cc:1199] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/a9672aab85c64d548a4f6583a1e9af20
I20260501 13:57:38.792982 1899 ts_tablet_manager.cc:1950] T a9672aab85c64d548a4f6583a1e9af20 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:38.793926 1629 raft_consensus.cc:2468] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 1.
I20260501 13:57:38.794826 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 164) took 1465 ms. Trace:
I20260501 13:57:38.795039 1719 leader_election.cc:304] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:38.774845 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "920d7f9748ae403d90573d612feeba59" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:38.796720 1882 raft_consensus.cc:359] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.797304 1882 raft_consensus.cc:385] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.798120 2047 log.cc:1199] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/adc47b3b2b084d33a12bd28cfad27382
I20260501 13:57:38.799157 2047 ts_tablet_manager.cc:1950] T adc47b3b2b084d33a12bd28cfad27382 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:38.797521 1882 raft_consensus.cc:740] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.804445 1882 consensus_queue.cc:260] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.808445 1881 tablet_bootstrap.cc:492] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:38.808979 1881 ts_tablet_manager.cc:1403] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.098s user 0.011s sys 0.000s
I20260501 13:57:38.813813 1882 ts_tablet_manager.cc:1434] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.025s user 0.004s sys 0.002s
W20260501 13:57:38.797497 1717 leader_election.cc:343] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.820345 1882 tablet_bootstrap.cc:492] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:38.798655 2282 raft_consensus.cc:2804] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:38.821645 2282 raft_consensus.cc:697] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Running, Role: LEADER
I20260501 13:57:38.822546 2282 consensus_queue.cc:237] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.827148 1883 tablet_bootstrap.cc:654] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:38.827492 1881 raft_consensus.cc:359] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.828226 1881 raft_consensus.cc:385] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.828485 1881 raft_consensus.cc:740] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.795113 1899 rpcz_store.cc:276] 0501 13:57:37.329037 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.329149 (+ 112us) service_pool.cc:224] Handling call
0501 13:57:38.688303 (+1359154us) ts_tablet_manager.cc:1151] Deleting tablet a9672aab85c64d548a4f6583a1e9af20
0501 13:57:38.688340 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:38.762569 (+ 74229us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:38.791885 (+ 29316us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:38.794767 (+ 2882us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1211309}
I20260501 13:57:38.831331 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet a9672aab85c64d548a4f6583a1e9af20 (table default.Table_bc1aee595536420799f8a28881253c09 [id=787286aa1fb44c45827992c1cf240e2d]) successfully deleted
I20260501 13:57:38.831811 1899 tablet_replica.cc:333] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:38.832490 1899 raft_consensus.cc:2243] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:38.832945 1899 raft_consensus.cc:2272] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:38.833273 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet adc47b3b2b084d33a12bd28cfad27382 (table default.Table_c4edbd4ba4c54b1e9f9909c9badd7c54 [id=1de866dfd0d145e7a13e5ae1f367524b]) successfully deleted
W20260501 13:57:38.835139 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 19c1e0c30dce488baed9e046d90f0ba4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19c1e0c30dce488baed9e046d90f0ba4 already in progress: creating tablet
W20260501 13:57:38.838323 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 already in progress: creating tablet
I20260501 13:57:38.840121 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_59bb2c3a0a8247d98ad73d3a2a21b1d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:38.840215 1881 consensus_queue.cc:260] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:38.780051 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 already in progress: creating tablet
W20260501 13:57:38.841259 1119 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 61244.4us
I20260501 13:57:38.842109 1123 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_46b8725e01684c90bcbbde7b5a9f4e0e" } new_table_name: "default.Table_7b8be07c742f48d0af3422dfea143dd1" modify_external_catalogs: true
I20260501 13:57:38.844234 1882 tablet_bootstrap.cc:654] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:38.845494 1883 tablet_bootstrap.cc:492] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
W20260501 13:57:38.846570 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59bb2c3a0a8247d98ad73d3a2a21b1d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:38.859836 1123 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 consensus state has no leader
I20260501 13:57:38.862744 1883 ts_tablet_manager.cc:1403] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.107s user 0.010s sys 0.003s
I20260501 13:57:38.865747 1883 raft_consensus.cc:359] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:38.874228 1421 consensus_peers.cc:597] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e41b43329e054442bc8697cc28161954. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:38.875420 1899 ts_tablet_manager.cc:1916] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:38.876820 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 already in progress: creating tablet
I20260501 13:57:38.866374 1883 raft_consensus.cc:385] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.882746 1883 raft_consensus.cc:740] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.884377 1881 ts_tablet_manager.cc:1434] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.075s user 0.005s sys 0.000s
I20260501 13:57:38.889386 1883 consensus_queue.cc:260] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.904547 1883 ts_tablet_manager.cc:1434] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.041s user 0.005s sys 0.001s
I20260501 13:57:38.904860 1881 tablet_bootstrap.cc:492] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:38.907977 1882 tablet_bootstrap.cc:492] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:38.908373 1882 ts_tablet_manager.cc:1403] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.088s user 0.005s sys 0.005s
I20260501 13:57:38.910643 1882 raft_consensus.cc:359] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.911252 1882 raft_consensus.cc:385] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:38.911491 1882 raft_consensus.cc:740] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:38.923403 1882 consensus_queue.cc:260] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.930287 1883 tablet_bootstrap.cc:492] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:38.938541 1882 ts_tablet_manager.cc:1434] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.030s user 0.005s sys 0.002s
I20260501 13:57:38.953052 1882 tablet_bootstrap.cc:492] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:38.955231 1881 tablet_bootstrap.cc:654] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:38.962910 1883 tablet_bootstrap.cc:654] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:38.968262 2273 raft_consensus.cc:493] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:38.968688 2273 raft_consensus.cc:515] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:38.969444 1117 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 (table default.Table_7b8be07c742f48d0af3422dfea143dd1 [id=39aeda29815d474196136813b5d811e7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 consensus state has no leader
I20260501 13:57:38.970305 1899 ts_tablet_manager.cc:1929] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:38.970691 1899 log.cc:1199] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/b98187c7843e42b09d55c851117c4ac8
I20260501 13:57:38.970978 2273 leader_election.cc:290] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:38.971280 1899 ts_tablet_manager.cc:1950] T b98187c7843e42b09d55c851117c4ac8 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:38.972357 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c789ca9e3c748d5bc79f4e1b0ea8fd1" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:38.972941 1782 raft_consensus.cc:2468] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:38.973090 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 163) took 1663 ms. Trace:
I20260501 13:57:38.973529 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c789ca9e3c748d5bc79f4e1b0ea8fd1" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:38.973373 1899 rpcz_store.cc:276] 0501 13:57:37.309305 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.309422 (+ 117us) service_pool.cc:224] Handling call
0501 13:57:38.831661 (+1522239us) ts_tablet_manager.cc:1151] Deleting tablet b98187c7843e42b09d55c851117c4ac8
0501 13:57:38.831706 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:38.909488 (+ 77782us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:38.970183 (+ 60695us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:38.973033 (+ 2850us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1340738}
I20260501 13:57:38.974414 1563 leader_election.cc:304] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:38.974618 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet b98187c7843e42b09d55c851117c4ac8 (table default.Table_8999f52c517a462a9845800aba73657e [id=e1549f4369a14aceb962e2be6f1a2ea4]) successfully deleted
I20260501 13:57:38.975205 2273 raft_consensus.cc:2804] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:38.975515 2273 raft_consensus.cc:493] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:38.975800 2273 raft_consensus.cc:3060] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:38.975955 1564 leader_election.cc:343] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:38.978646 1881 tablet_bootstrap.cc:492] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:38.979048 1881 ts_tablet_manager.cc:1403] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.000s
I20260501 13:57:38.981069 1124 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:38.981755 1129 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c9c10d0b01f846849dcd931a75ff0d63" } modify_external_catalogs: true
I20260501 13:57:38.982230 1129 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c9c10d0b01f846849dcd931a75ff0d63" } modify_external_catalogs: true
I20260501 13:57:38.982033 1881 raft_consensus.cc:359] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:38.985782 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_6ab4c80251da40a783ac60f1b02b941d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:38.985553 2273 raft_consensus.cc:515] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:38.987224 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ab4c80251da40a783ac60f1b02b941d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:38.988621 1127 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_ad4c78cb126f4013968e7b0e20a5c47c [id=0f9c1230865848adaa4e5ea744c31c8b] alter complete (version 4)
I20260501 13:57:38.989132 1126 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_360decd489b74f4b8ee93b008e57c45b [id=8cff6f6a2ebc447c979a45c9bfa438dc] alter complete (version 2)
I20260501 13:57:38.989585 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c789ca9e3c748d5bc79f4e1b0ea8fd1" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:38.990116 1782 raft_consensus.cc:3060] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:38.994043 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_d56638cc43fa42f1a0da405e69383a6a" } new_table_name: "default.Table_d986c7fb196a4dc2a25b8bba537072c9" modify_external_catalogs: true
I20260501 13:57:38.995491 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c789ca9e3c748d5bc79f4e1b0ea8fd1" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:38.997398 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_c32278ab7552423e922a3248d158e499"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:38.998811 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c32278ab7552423e922a3248d158e499 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:38.999796 1564 leader_election.cc:343] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.001274 1782 raft_consensus.cc:2468] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:39.002250 1563 leader_election.cc:304] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:39.003036 2274 raft_consensus.cc:2804] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:39.004673 2282 raft_consensus.cc:493] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.005131 2282 raft_consensus.cc:515] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.007002 2282 leader_election.cc:290] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:39.009049 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd02738e5f4ecfbfa06d5541614ffd" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:39.009220 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 77c59fc1fdd645af8c3017b6836c8bf4 (DEFAULT_TABLE table=default.Table_6bab24ce4e874b7abcaf569c239b41ea [id=7829eeffca1d405ba5dbe7b4c94da149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.010165 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c59fc1fdd645af8c3017b6836c8bf4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:39.010313 1717 leader_election.cc:343] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.012068 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 77c59fc1fdd645af8c3017b6836c8bf4 (DEFAULT_TABLE table=default.Table_6bab24ce4e874b7abcaf569c239b41ea [id=7829eeffca1d405ba5dbe7b4c94da149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.013084 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c59fc1fdd645af8c3017b6836c8bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.015568 1882 tablet_bootstrap.cc:654] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:38.982645 1881 raft_consensus.cc:385] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.018635 1881 raft_consensus.cc:740] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.019415 2273 leader_election.cc:290] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:39.019379 1881 consensus_queue.cc:260] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.019876 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet 104708eb7fc0412dad0bd7e14eee9d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.020547 2093 tablet_replica.cc:333] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:39.021221 2093 raft_consensus.cc:2243] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:39.021641 2093 raft_consensus.cc:2272] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:39.023278 2093 ts_tablet_manager.cc:1916] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:39.025535 1761 tablet_service.cc:1511] Processing CreateTablet for tablet d5e1ab4acf0d400b963c3c61772f6272 (DEFAULT_TABLE table=default.Table_1d160b6826e74a69979a8b8500b8ca06 [id=8957f416d90c4db1b85886c2c43d7870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.026290 2274 raft_consensus.cc:697] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:39.026548 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e1ab4acf0d400b963c3c61772f6272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.026707 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd02738e5f4ecfbfa06d5541614ffd" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:39.028370 1881 ts_tablet_manager.cc:1434] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20260501 13:57:39.029268 1881 tablet_bootstrap.cc:492] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:39.027449 2274 consensus_queue.cc:237] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.030908 1461 tablet_service.cc:1511] Processing CreateTablet for tablet d5e1ab4acf0d400b963c3c61772f6272 (DEFAULT_TABLE table=default.Table_1d160b6826e74a69979a8b8500b8ca06 [id=8957f416d90c4db1b85886c2c43d7870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.031848 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e1ab4acf0d400b963c3c61772f6272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.036636 1882 tablet_bootstrap.cc:492] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:39.039723 1882 ts_tablet_manager.cc:1403] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.000s
I20260501 13:57:39.041769 1603 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ff05404061447199d936d005afda653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.041914 1882 raft_consensus.cc:359] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.042546 1882 raft_consensus.cc:385] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.042784 1882 raft_consensus.cc:740] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.042954 1603 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ee580bb9d3843f9bbeae8eef2c347ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.042929 2218 raft_consensus.cc:493] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.043342 2218 raft_consensus.cc:515] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.043921 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 77c59fc1fdd645af8c3017b6836c8bf4 (DEFAULT_TABLE table=default.Table_6bab24ce4e874b7abcaf569c239b41ea [id=7829eeffca1d405ba5dbe7b4c94da149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.044833 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c59fc1fdd645af8c3017b6836c8bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.046110 2218 leader_election.cc:290] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:39.047497 1418 leader_election.cc:343] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 104708eb7fc0412dad0bd7e14eee9d13
I20260501 13:57:39.048890 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 176) took 1207 ms. Trace:
I20260501 13:57:39.049223 1899 rpcz_store.cc:276] 0501 13:57:37.841140 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:37.841253 (+ 113us) service_pool.cc:224] Handling call
0501 13:57:39.048546 (+1207293us) ts_tablet_manager.cc:1151] Deleting tablet 1042bdf58c1f488daeaec284eb3319d5
0501 13:57:39.048601 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:39.048834 (+ 233us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206708}
W20260501 13:57:39.051092 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 1042bdf58c1f488daeaec284eb3319d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1042bdf58c1f488daeaec284eb3319d5 already in progress: creating tablet
I20260501 13:57:39.054692 1882 consensus_queue.cc:260] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.055501 1603 tablet_service.cc:1511] Processing CreateTablet for tablet d5e1ab4acf0d400b963c3c61772f6272 (DEFAULT_TABLE table=default.Table_1d160b6826e74a69979a8b8500b8ca06 [id=8957f416d90c4db1b85886c2c43d7870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.056499 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e1ab4acf0d400b963c3c61772f6272. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:39.059026 1719 leader_election.cc:343] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.059389 1719 leader_election.cc:304] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:39.060009 2282 raft_consensus.cc:2749] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:39.062229 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 19c1e0c30dce488baed9e046d90f0ba4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19c1e0c30dce488baed9e046d90f0ba4 already in progress: creating tablet
I20260501 13:57:39.063270 1755 tablet_service.cc:1511] Processing CreateTablet for tablet 53d9a8503c8d4bebb89ea493b99bf6fd (DEFAULT_TABLE table=default.Table_4e1547b16dcc4d878bcc141802b827dc [id=383d0757f1654b0b8de592bc1e1c3071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.063529 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_db50865558a14081b6f0280d9f45ceea" } new_table_name: "default.Table_536f8eeddbae410cbc4414090433de66" modify_external_catalogs: true
I20260501 13:57:39.064375 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d9a8503c8d4bebb89ea493b99bf6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.064759 1882 ts_tablet_manager.cc:1434] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260501 13:57:39.065613 1882 tablet_bootstrap.cc:492] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:39.069008 1753 tablet_service.cc:1511] Processing CreateTablet for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd (DEFAULT_TABLE table=default.Table_d23b951fef3048af97eb045b842ebd22 [id=d1895d92bfea4a13be9ab6e5454da8d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.070051 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.072919 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 53d9a8503c8d4bebb89ea493b99bf6fd (DEFAULT_TABLE table=default.Table_4e1547b16dcc4d878bcc141802b827dc [id=383d0757f1654b0b8de592bc1e1c3071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.074376 1463 tablet_service.cc:1511] Processing CreateTablet for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd (DEFAULT_TABLE table=default.Table_d23b951fef3048af97eb045b842ebd22 [id=d1895d92bfea4a13be9ab6e5454da8d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.077162 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "104708eb7fc0412dad0bd7e14eee9d13" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:39.080238 1883 tablet_bootstrap.cc:492] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:39.080693 1883 ts_tablet_manager.cc:1403] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.151s user 0.006s sys 0.005s
I20260501 13:57:39.085788 1899 tablet_replica.cc:333] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:39.086961 1899 raft_consensus.cc:2243] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:39.087482 1899 raft_consensus.cc:2272] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:39.087786 1882 tablet_bootstrap.cc:654] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:39.092767 1881 tablet_bootstrap.cc:654] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:39.099390 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_059ab1f09d8443f3a261ec50523dbe80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:39.100909 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_059ab1f09d8443f3a261ec50523dbe80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:39.102835 1882 tablet_bootstrap.cc:492] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:39.103216 1882 ts_tablet_manager.cc:1403] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.000s
I20260501 13:57:39.105449 1882 raft_consensus.cc:359] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.106048 1882 raft_consensus.cc:385] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.106290 1882 raft_consensus.cc:740] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.106920 1882 consensus_queue.cc:260] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.093681 1883 raft_consensus.cc:359] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.108341 1883 raft_consensus.cc:385] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.108644 1883 raft_consensus.cc:740] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.108994 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 15a1f80cf7d24974b00c6cca7ae5783e (DEFAULT_TABLE table=default.Table_7bbaf820afc746198b26a791c87cb736 [id=7bde73227e33452984fc7154a5547299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.111290 1882 ts_tablet_manager.cc:1434] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260501 13:57:39.114264 1882 tablet_bootstrap.cc:492] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:39.115191 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a1f80cf7d24974b00c6cca7ae5783e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.119704 1882 tablet_bootstrap.cc:654] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:39.121835 1610 tablet_service.cc:1511] Processing CreateTablet for tablet 53d9a8503c8d4bebb89ea493b99bf6fd (DEFAULT_TABLE table=default.Table_4e1547b16dcc4d878bcc141802b827dc [id=383d0757f1654b0b8de592bc1e1c3071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.122974 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d9a8503c8d4bebb89ea493b99bf6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.126611 1753 tablet_service.cc:1511] Processing CreateTablet for tablet f40c836991034c7bac14069c63a93955 (DEFAULT_TABLE table=default.Table_68ae6dc027ee4521baf488d0c406a1e1 [id=fc41989612d543a69234baf8bcbfbbcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.127497 1602 tablet_service.cc:1511] Processing CreateTablet for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd (DEFAULT_TABLE table=default.Table_d23b951fef3048af97eb045b842ebd22 [id=d1895d92bfea4a13be9ab6e5454da8d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.133917 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.132716 1457 tablet_service.cc:1511] Processing CreateTablet for tablet 0e2953dd57af41ea98d87d299e1a5f12 (DEFAULT_TABLE table=default.Table_d132a7668b34479887548c901967b36c [id=2fb661abb47848e2b4a7df024ecb18f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.104107 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 15a1f80cf7d24974b00c6cca7ae5783e (DEFAULT_TABLE table=default.Table_7bbaf820afc746198b26a791c87cb736 [id=7bde73227e33452984fc7154a5547299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.113905 1462 tablet_service.cc:1511] Processing CreateTablet for tablet f40c836991034c7bac14069c63a93955 (DEFAULT_TABLE table=default.Table_68ae6dc027ee4521baf488d0c406a1e1 [id=fc41989612d543a69234baf8bcbfbbcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.116802 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53d9a8503c8d4bebb89ea493b99bf6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.142716 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.143726 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f40c836991034c7bac14069c63a93955. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:39.095919 1421 leader_election.cc:343] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: Tablet not RUNNING: STOPPED
I20260501 13:57:39.146832 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a1f80cf7d24974b00c6cca7ae5783e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.146971 1421 leader_election.cc:304] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:39.147581 2218 raft_consensus.cc:2749] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:39.148298 1421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 52436.1us
I20260501 13:57:39.150924 1457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2953dd57af41ea98d87d299e1a5f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.157850 1881 tablet_bootstrap.cc:492] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:39.158272 1881 ts_tablet_manager.cc:1403] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.000s
I20260501 13:57:39.159345 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_b4da3389e7934799aa303ed608a26082"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:39.160785 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4da3389e7934799aa303ed608a26082 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:39.161454 1881 raft_consensus.cc:359] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.162029 1881 raft_consensus.cc:385] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.162245 1881 raft_consensus.cc:740] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.165802 1881 consensus_queue.cc:260] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.173707 1881 ts_tablet_manager.cc:1434] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260501 13:57:39.174618 1881 tablet_bootstrap.cc:492] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:39.199365 1881 tablet_bootstrap.cc:654] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:39.109328 1883 consensus_queue.cc:260] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.213975 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f40c836991034c7bac14069c63a93955. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.214757 1883 ts_tablet_manager.cc:1434] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.134s user 0.007s sys 0.001s
I20260501 13:57:39.215651 1883 tablet_bootstrap.cc:492] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:39.131395 1899 ts_tablet_manager.cc:1916] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:39.186546 2282 raft_consensus.cc:493] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.226877 2282 raft_consensus.cc:515] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.228718 2282 leader_election.cc:290] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:39.222353 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 691464fc50314a498a5399601b43c733 (DEFAULT_TABLE table=default.Table_761ae80bcd5e4b1ebde69f67afb292eb [id=7a31799eed93403db75f46e33e791c99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.230969 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691464fc50314a498a5399601b43c733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.231808 1755 tablet_service.cc:1511] Processing CreateTablet for tablet 0e2953dd57af41ea98d87d299e1a5f12 (DEFAULT_TABLE table=default.Table_d132a7668b34479887548c901967b36c [id=2fb661abb47848e2b4a7df024ecb18f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.232877 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2953dd57af41ea98d87d299e1a5f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.236002 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9c92846d4e4255a1c8ce3835a0a18d" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:39.237228 1717 leader_election.cc:343] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.252774 1883 tablet_bootstrap.cc:654] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:39.258301 1129 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 569a0240b1274d9bb830addbaf1ec343
I20260501 13:57:39.258325 1608 tablet_service.cc:1511] Processing CreateTablet for tablet f40c836991034c7bac14069c63a93955 (DEFAULT_TABLE table=default.Table_68ae6dc027ee4521baf488d0c406a1e1 [id=fc41989612d543a69234baf8bcbfbbcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.259459 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f40c836991034c7bac14069c63a93955. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.260511 2274 raft_consensus.cc:493] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.260924 2274 raft_consensus.cc:515] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.261965 1602 tablet_service.cc:1511] Processing CreateTablet for tablet 15a1f80cf7d24974b00c6cca7ae5783e (DEFAULT_TABLE table=default.Table_7bbaf820afc746198b26a791c87cb736 [id=7bde73227e33452984fc7154a5547299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.262732 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet 569a0240b1274d9bb830addbaf1ec343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.262923 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a1f80cf7d24974b00c6cca7ae5783e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.263939 1131 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 8d2f196f44be4fbc8059901380b71271
I20260501 13:57:39.264066 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 569a0240b1274d9bb830addbaf1ec343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:39.264261 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 569a0240b1274d9bb830addbaf1ec343 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 569a0240b1274d9bb830addbaf1ec343 already in progress: creating tablet
I20260501 13:57:39.265059 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 0e2953dd57af41ea98d87d299e1a5f12 (DEFAULT_TABLE table=default.Table_d132a7668b34479887548c901967b36c [id=2fb661abb47848e2b4a7df024ecb18f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.265568 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 691464fc50314a498a5399601b43c733 (DEFAULT_TABLE table=default.Table_761ae80bcd5e4b1ebde69f67afb292eb [id=7a31799eed93403db75f46e33e791c99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.265980 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2953dd57af41ea98d87d299e1a5f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.266433 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691464fc50314a498a5399601b43c733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.267026 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d2f196f44be4fbc8059901380b71271 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.268972 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9c92846d4e4255a1c8ce3835a0a18d" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:39.270641 1127 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.129:53385 (request call id 55) took 965 ms (client timeout 1000 ms). Trace:
W20260501 13:57:39.270948 1127 rpcz_store.cc:269] 0501 13:57:38.304871 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:38.304991 (+ 120us) service_pool.cc:224] Handling call
0501 13:57:39.270580 (+965589us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:38.547672 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:38.548035 (+ 363us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:38.548056 (+ 21us) write_op.cc:435] Acquired schema lock
0501 13:57:38.548067 (+ 11us) tablet.cc:662] Decoding operations
0501 13:57:38.548300 (+ 233us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:38.548333 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:38.548356 (+ 23us) tablet.cc:685] Acquiring locks for 1 operations
0501 13:57:38.548487 (+ 131us) tablet.cc:701] Row locks acquired
0501 13:57:38.548500 (+ 13us) write_op.cc:260] PREPARE: finished
0501 13:57:38.548604 (+ 104us) write_op.cc:270] Start()
0501 13:57:38.548688 (+ 84us) write_op.cc:276] Timestamp: P: 1777643858548586 usec, L: 0
0501 13:57:38.548702 (+ 14us) op_driver.cc:348] REPLICATION: starting
0501 13:57:38.602707 (+ 54005us) log.cc:844] Serialized 356 byte log entry
0501 13:57:38.603440 (+ 733us) spinlock_profiling.cc:243] Waited 53.5 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663024951 00007f56630b7208 00007f56635299f1 00007f56635291c8 00007f566352de61 00007f566352de19 00007f566352dda9 00007f566352dd71 00007f566352cf9d 00000000005b3bc4 00000000005b39f9
0501 13:57:38.676344 (+ 72904us) op_driver.cc:464] REPLICATION: finished
0501 13:57:38.676813 (+ 469us) write_op.cc:301] APPLY: starting
0501 13:57:38.676871 (+ 58us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:38.676952 (+ 81us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:38.676971 (+ 19us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:38.677143 (+ 172us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:38.677268 (+ 125us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:57:38.677310 (+ 42us) write_op.cc:312] APPLY: finished
0501 13:57:38.677480 (+ 170us) log.cc:844] Serialized 23 byte log entry
0501 13:57:38.678094 (+ 614us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:38.678154 (+ 60us) write_op.cc:454] Released schema lock
0501 13:57:38.678183 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:38.678390 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:38.678417 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":311,"apply.run_cpu_time_us":1102,"apply.run_wall_time_us":1559,"num_ops":1,"prepare.queue_time_us":93,"prepare.run_cpu_time_us":3054,"prepare.run_wall_time_us":55992,"raft.queue_time_us":482,"raft.run_cpu_time_us":617,"raft.run_wall_time_us":616,"replication_time_us":127551,"spinlock_wait_cycles":123085568}]]}
I20260501 13:57:39.276067 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d2f196f44be4fbc8059901380b71271 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.276854 2274 leader_election.cc:290] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:39.278610 2047 tablet_replica.cc:333] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
W20260501 13:57:39.278699 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b125d1a190f947a5b2ef942f04408c51 consensus state has no leader
I20260501 13:57:39.278638 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 691464fc50314a498a5399601b43c733 (DEFAULT_TABLE table=default.Table_761ae80bcd5e4b1ebde69f67afb292eb [id=7a31799eed93403db75f46e33e791c99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.279271 2047 raft_consensus.cc:2243] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:39.279588 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691464fc50314a498a5399601b43c733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.280078 2047 raft_consensus.cc:2272] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:39.280280 1899 ts_tablet_manager.cc:1929] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:39.280608 1899 log.cc:1199] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/9c789ca9e3c748d5bc79f4e1b0ea8fd1
I20260501 13:57:39.281183 1899 ts_tablet_manager.cc:1950] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:39.281875 2047 ts_tablet_manager.cc:1916] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:39.282843 1719 leader_election.cc:343] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:39.283051 1563 leader_election.cc:343] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: b507a4d4c9c54bc68d92d9286794a264
I20260501 13:57:39.283211 1719 leader_election.cc:304] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:39.284291 2282 raft_consensus.cc:2749] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:39.284665 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a563bccc7b046658ed133109f676528 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.290073 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ff05404061447199d936d005afda653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:39.296263 1124 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.131:42895 (request call id 59) took 1003 ms (client timeout 1000 ms). Trace:
I20260501 13:57:39.296381 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet ec4bd4cb1e70472aa09fc93afc3b1501 with delete_type TABLET_DATA_DELETED (replaced by tablet 8dbb619863dd405ab20ead6bbebf4d69) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.303711 1752 tablet_service.cc:1511] Processing CreateTablet for tablet 3223417aa5594ef7942c6d90bdc8711c (DEFAULT_TABLE table=default.Table_f8e9d88347b54e2da903776319f1c181 [id=b3e0852bf1bc4eedad8b69f7e4cf0785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.304315 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 8503ad295c0847ae9c4fcf2c97aee69e (DEFAULT_TABLE table=default.Table_d04310990233457bbf23e88ca7321db7 [id=fd45a3ba78a34ade9f53e64889543ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.304883 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3223417aa5594ef7942c6d90bdc8711c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.305617 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 3223417aa5594ef7942c6d90bdc8711c (DEFAULT_TABLE table=default.Table_f8e9d88347b54e2da903776319f1c181 [id=b3e0852bf1bc4eedad8b69f7e4cf0785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.306928 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet e41b43329e054442bc8697cc28161954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:29 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.314162 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet b507a4d4c9c54bc68d92d9286794a264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.314718 1460 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ee580bb9d3843f9bbeae8eef2c347ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:30 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.314710 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet 104708eb7fc0412dad0bd7e14eee9d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.320391 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b507a4d4c9c54bc68d92d9286794a264" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:39.320992 1483 raft_consensus.cc:2468] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:39.321806 2282 raft_consensus.cc:493] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.322227 2282 raft_consensus.cc:515] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.324033 2282 leader_election.cc:290] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:39.325994 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67b0675398d04be08bd29638b4ba2083" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:39.327754 1717 leader_election.cc:343] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.298091 1130 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 0669a229d6d54eacac9889ad18e06f30
I20260501 13:57:39.331514 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67b0675398d04be08bd29638b4ba2083" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:39.332290 1130 catalog_manager.cc:6741] ReplaceTablet: tablet 0669a229d6d54eacac9889ad18e06f30 deleted and replaced by tablet 81230beeff8d434aa2dac467c35d9703
I20260501 13:57:39.333914 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 0669a229d6d54eacac9889ad18e06f30 with delete_type TABLET_DATA_DELETED (replaced by tablet 81230beeff8d434aa2dac467c35d9703) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:39.335072 1124 rpcz_store.cc:269] 0501 13:57:38.293327 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:38.295609 (+ 2282us) service_pool.cc:224] Handling call
0501 13:57:39.296187 (+1000578us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":7}
I20260501 13:57:39.338263 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet 0669a229d6d54eacac9889ad18e06f30 with delete_type TABLET_DATA_DELETED (replaced by tablet 81230beeff8d434aa2dac467c35d9703) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:39.338377 1719 leader_election.cc:343] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.338765 1719 leader_election.cc:304] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:39.339389 2282 raft_consensus.cc:2749] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:39.344092 1564 leader_election.cc:304] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:39.344796 2274 raft_consensus.cc:2804] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:39.345118 2274 raft_consensus.cc:493] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:39.345427 2274 raft_consensus.cc:3060] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:39.348063 1883 tablet_bootstrap.cc:492] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:39.348472 1883 ts_tablet_manager.cc:1403] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.133s user 0.004s sys 0.005s
I20260501 13:57:39.351604 2274 raft_consensus.cc:515] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:39.355197 1563 leader_election.cc:343] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: b507a4d4c9c54bc68d92d9286794a264
I20260501 13:57:39.355803 2274 leader_election.cc:290] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:39.355785 1119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.188:38815, user_credentials={real_user=slave}} blocked reactor thread for 106461us
I20260501 13:57:39.359200 1882 tablet_bootstrap.cc:492] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:39.359572 1882 ts_tablet_manager.cc:1403] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.245s user 0.047s sys 0.022s
I20260501 13:57:39.361943 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d2f196f44be4fbc8059901380b71271 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.363103 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 3223417aa5594ef7942c6d90bdc8711c (DEFAULT_TABLE table=default.Table_f8e9d88347b54e2da903776319f1c181 [id=b3e0852bf1bc4eedad8b69f7e4cf0785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.363185 1601 tablet_service.cc:1511] Processing CreateTablet for tablet 8503ad295c0847ae9c4fcf2c97aee69e (DEFAULT_TABLE table=default.Table_d04310990233457bbf23e88ca7321db7 [id=fd45a3ba78a34ade9f53e64889543ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.364085 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3223417aa5594ef7942c6d90bdc8711c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.364645 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8503ad295c0847ae9c4fcf2c97aee69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.369675 1130 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_68ae6dc027ee4521baf488d0c406a1e1" } new_table_name: "default.Table_cef2c2ac1c834f1bb056176a45d17cee" modify_external_catalogs: true
I20260501 13:57:39.374866 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 0669a229d6d54eacac9889ad18e06f30 with delete_type TABLET_DATA_DELETED (replaced by tablet 81230beeff8d434aa2dac467c35d9703) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.374637 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_5fba8e9c07514a4babeda53e157a580f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:39.376019 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fba8e9c07514a4babeda53e157a580f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:39.377887 1881 tablet_bootstrap.cc:492] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:39.378284 1881 ts_tablet_manager.cc:1403] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.204s user 0.007s sys 0.012s
I20260501 13:57:39.378985 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_ec911a9efe2c4a70a032be2376620b96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:39.380605 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec911a9efe2c4a70a032be2376620b96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:39.380605 1881 raft_consensus.cc:359] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.381235 1881 raft_consensus.cc:385] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.381475 1881 raft_consensus.cc:740] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.387581 1883 raft_consensus.cc:359] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.388775 1883 raft_consensus.cc:385] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.389228 1883 raft_consensus.cc:740] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.390415 1883 consensus_queue.cc:260] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:39.395223 1130 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f40c836991034c7bac14069c63a93955 consensus state has no leader
I20260501 13:57:39.402686 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 (table default.Table_2a1ca2e987e243ad9f1c064a28866dd3 [id=ae0dad3b628b454d899e4912c8013d14]) successfully deleted
I20260501 13:57:39.406782 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8503ad295c0847ae9c4fcf2c97aee69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.407349 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3223417aa5594ef7942c6d90bdc8711c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.418699 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 95ed6155f481446780007300ec19e2c8 (DEFAULT_TABLE table=default.Table_62888a4c57cb4690bf1ab86c99529aa4 [id=8a57c622e594467882d2cec57a9fa46e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.420145 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 8503ad295c0847ae9c4fcf2c97aee69e (DEFAULT_TABLE table=default.Table_d04310990233457bbf23e88ca7321db7 [id=fd45a3ba78a34ade9f53e64889543ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.421121 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8503ad295c0847ae9c4fcf2c97aee69e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:39.424175 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 already in progress: creating tablet
W20260501 13:57:39.424818 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 569a0240b1274d9bb830addbaf1ec343 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 569a0240b1274d9bb830addbaf1ec343 already in progress: creating tablet
I20260501 13:57:39.425026 1899 tablet_replica.cc:333] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:39.425767 1899 raft_consensus.cc:2243] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260501 13:57:39.427145 1119 consensus_peers.cc:597] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b -> Peer 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815): Couldn't send request to peer 50c0932256e74399a537670bb424a1b7. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260501 13:57:39.430765 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b507a4d4c9c54bc68d92d9286794a264" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:39.431787 1483 raft_consensus.cc:3060] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:39.435577 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ed6155f481446780007300ec19e2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.438361 1483 raft_consensus.cc:2468] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:39.442127 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 920d7f9748ae403d90573d612feeba59 (table default.Table_f0801819716e426f92844203c4a06bdc [id=068f5ac17d494d0b8639cbdc07a04597]). Expected version 2 got 0
I20260501 13:57:39.442665 1457 tablet_service.cc:1511] Processing CreateTablet for tablet 069f9e9b9b6c40d48c9a622b73e83dec (DEFAULT_TABLE table=default.Table_76959fdf8538441aa44f9e120288cdab [id=7803add44edc41d98147ce2f7ba60ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.442879 1126 catalog_manager.cc:5671] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:39.443698 1457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069f9e9b9b6c40d48c9a622b73e83dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.443987 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet c33a1f423bdb44f5843daffe8f73e74a (table default.Table_ecd46a159d2448f9ba09fc99e14a22a3 [id=d98d5437c5bc4e44bcfe27983dfa36e6]). Expected version 1 got 0
I20260501 13:57:39.451224 1755 tablet_service.cc:1511] Processing CreateTablet for tablet 95ed6155f481446780007300ec19e2c8 (DEFAULT_TABLE table=default.Table_62888a4c57cb4690bf1ab86c99529aa4 [id=8a57c622e594467882d2cec57a9fa46e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.452263 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ed6155f481446780007300ec19e2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.453423 1883 ts_tablet_manager.cc:1434] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.105s user 0.000s sys 0.007s
I20260501 13:57:39.456630 1899 raft_consensus.cc:2272] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:39.406287 2318 tablet.cc:1724] T d2c2873572de4889a16fda8fe31ad594 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260501 13:57:39.427050 2319 tablet.cc:1724] T d2c2873572de4889a16fda8fe31ad594 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260501 13:57:39.401302 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet 569a0240b1274d9bb830addbaf1ec343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.391439 2093 ts_tablet_manager.cc:1929] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:39.460311 2093 log.cc:1199] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/104708eb7fc0412dad0bd7e14eee9d13
I20260501 13:57:39.460774 1883 tablet_bootstrap.cc:492] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:39.461045 2093 ts_tablet_manager.cc:1950] T 104708eb7fc0412dad0bd7e14eee9d13 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:39.439672 1564 leader_election.cc:304] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:39.466325 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 95ed6155f481446780007300ec19e2c8 (DEFAULT_TABLE table=default.Table_62888a4c57cb4690bf1ab86c99529aa4 [id=8a57c622e594467882d2cec57a9fa46e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.456496 1882 raft_consensus.cc:359] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.467060 1882 raft_consensus.cc:385] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.467314 1882 raft_consensus.cc:740] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.467443 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ed6155f481446780007300ec19e2c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:39.426366 1563 consensus_peers.cc:597] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Couldn't send request to peer 0ed1f9a545474e76a661c32bf11a7fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ff05404061447199d936d005afda653. This is attempt 1: this message will repeat every 5th retry.
W20260501 13:57:39.470952 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 44789.2us
W20260501 13:57:39.471388 2317 log.cc:927] Time spent T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260501 13:57:39.474431 2302 consensus_queue.cc:1048] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:39.474786 2282 raft_consensus.cc:493] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.475237 2282 raft_consensus.cc:515] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.476399 1899 ts_tablet_manager.cc:1916] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:39.477061 2282 leader_election.cc:290] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:39.477150 1463 tablet_service.cc:1511] Processing CreateTablet for tablet cfce57b777094ff0a135bd4f92d5f7dc (DEFAULT_TABLE table=default.Table_3689c91dc5a941d0a884485d780a9ef4 [id=0a1dc33d7b424d12a18740bc2c22676a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.477504 1601 tablet_service.cc:1511] Processing CreateTablet for tablet 069f9e9b9b6c40d48c9a622b73e83dec (DEFAULT_TABLE table=default.Table_76959fdf8538441aa44f9e120288cdab [id=7803add44edc41d98147ce2f7ba60ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.478116 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfce57b777094ff0a135bd4f92d5f7dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.478432 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069f9e9b9b6c40d48c9a622b73e83dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.470744 2274 raft_consensus.cc:2804] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:39.480768 2274 raft_consensus.cc:697] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:39.481526 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0356b19f0aa4197a41cc66d4440bffb" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:39.474184 1753 tablet_service.cc:1511] Processing CreateTablet for tablet 069f9e9b9b6c40d48c9a622b73e83dec (DEFAULT_TABLE table=default.Table_76959fdf8538441aa44f9e120288cdab [id=7803add44edc41d98147ce2f7ba60ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.467870 1882 consensus_queue.cc:260] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:39.482784 1717 leader_election.cc:343] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.487191 1882 ts_tablet_manager.cc:1434] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.127s user 0.004s sys 0.002s
I20260501 13:57:39.487550 1131 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_da800013eb694abeba00d698e4bbc439" } modify_external_catalogs: true
I20260501 13:57:39.488050 1131 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_da800013eb694abeba00d698e4bbc439" } modify_external_catalogs: true
I20260501 13:57:39.491304 1124 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_220e55459a0a41c69abf3fd74bca7bdd" } modify_external_catalogs: true
I20260501 13:57:39.491819 1124 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_220e55459a0a41c69abf3fd74bca7bdd" } modify_external_catalogs: true
I20260501 13:57:39.498449 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0356b19f0aa4197a41cc66d4440bffb" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:39.499818 1719 leader_election.cc:343] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.500224 1719 leader_election.cc:304] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:39.492540 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_3e410e6cd57e44db8ad5287f606630fd" } new_table_name: "default.Table_a7b6759170be4cce860b79126e7bbc29" modify_external_catalogs: true
I20260501 13:57:39.501175 2282 raft_consensus.cc:2749] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:39.496483 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_bc6341b170b3425eb49b5cc95ed40eb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:39.504330 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc6341b170b3425eb49b5cc95ed40eb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:39.430733 2047 ts_tablet_manager.cc:1929] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:57:39.511332 2047 log.cc:1199] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/8d2f196f44be4fbc8059901380b71271
I20260501 13:57:39.512037 2047 ts_tablet_manager.cc:1950] T 8d2f196f44be4fbc8059901380b71271 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:39.513113 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069f9e9b9b6c40d48c9a622b73e83dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.513257 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 8d2f196f44be4fbc8059901380b71271 (table default.Table_360decd489b74f4b8ee93b008e57c45b [id=8cff6f6a2ebc447c979a45c9bfa438dc]) successfully deleted
I20260501 13:57:39.514124 2093 tablet_replica.cc:333] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:39.499799 1881 consensus_queue.cc:260] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.518745 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 104708eb7fc0412dad0bd7e14eee9d13 (table default.Table_4a57dcc1c3194af793a3a0367e766cc6 [id=a921f986b8ac4a948ccb297e1084fccb]) successfully deleted
I20260501 13:57:39.531239 1752 tablet_service.cc:1511] Processing CreateTablet for tablet cfce57b777094ff0a135bd4f92d5f7dc (DEFAULT_TABLE table=default.Table_3689c91dc5a941d0a884485d780a9ef4 [id=0a1dc33d7b424d12a18740bc2c22676a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.532413 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfce57b777094ff0a135bd4f92d5f7dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.535650 1131 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 68cb7bd1a5aa4419b9c5176a287fe885
I20260501 13:57:39.542634 1602 tablet_service.cc:1511] Processing CreateTablet for tablet cfce57b777094ff0a135bd4f92d5f7dc (DEFAULT_TABLE table=default.Table_3689c91dc5a941d0a884485d780a9ef4 [id=0a1dc33d7b424d12a18740bc2c22676a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:39.543980 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfce57b777094ff0a135bd4f92d5f7dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:39.544055 1883 tablet_bootstrap.cc:654] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:39.544278 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet 68cb7bd1a5aa4419b9c5176a287fe885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.546888 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_33fb06bbd21c4209919e7b4711c86e48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:39.548349 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33fb06bbd21c4209919e7b4711c86e48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:39.555202 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9018d70c734a429aa59565ad67a46acc consensus state has no leader
I20260501 13:57:39.562266 2218 raft_consensus.cc:493] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.562780 2218 raft_consensus.cc:515] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.565248 2218 leader_election.cc:290] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:39.567008 1418 leader_election.cc:343] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 2ee580bb9d3843f9bbeae8eef2c347ad
I20260501 13:57:39.573171 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet 68cb7bd1a5aa4419b9c5176a287fe885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.573819 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 1c630c25ec5c46539936ec545ff4d292
I20260501 13:57:39.576228 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet 1c630c25ec5c46539936ec545ff4d292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.577565 2093 raft_consensus.cc:2243] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:39.579831 1881 ts_tablet_manager.cc:1434] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.201s user 0.005s sys 0.000s
I20260501 13:57:39.580684 1881 tablet_bootstrap.cc:492] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:39.583228 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet 1c630c25ec5c46539936ec545ff4d292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.592451 2093 pending_rounds.cc:70] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: Trying to abort 1 pending ops.
I20260501 13:57:39.598616 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_8a77fe75f8044d75a97b45dd4d94216f" } new_table_name: "default.Table_392605e427d14287b4fccdd9b198c6a2" modify_external_catalogs: true
W20260501 13:57:39.601135 1117 catalog_manager.cc:4969] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 4a563bccc7b046658ed133109f676528 because the tablet was not found. No further retry: Not found: Tablet not found: 4a563bccc7b046658ed133109f676528
I20260501 13:57:39.614497 2047 tablet_replica.cc:333] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:39.615234 2047 raft_consensus.cc:2243] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:39.615670 2047 raft_consensus.cc:2272] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:39.617266 2047 ts_tablet_manager.cc:1916] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:39.619765 1883 tablet_bootstrap.cc:492] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:39.620154 1883 ts_tablet_manager.cc:1403] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.160s user 0.008s sys 0.003s
I20260501 13:57:39.621891 1899 ts_tablet_manager.cc:1929] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:57:39.622206 1899 log.cc:1199] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/8d2f196f44be4fbc8059901380b71271
I20260501 13:57:39.622732 1882 tablet_bootstrap.cc:492] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:39.622864 1899 ts_tablet_manager.cc:1950] T 8d2f196f44be4fbc8059901380b71271 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:39.627180 1883 raft_consensus.cc:359] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.592792 2093 pending_rounds.cc:77] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229244288450560 op_type: NO_OP noop_request { }
I20260501 13:57:39.627770 1883 raft_consensus.cc:385] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.628005 1883 raft_consensus.cc:740] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.628037 2093 raft_consensus.cc:2889] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:39.628340 2093 raft_consensus.cc:2272] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:39.628650 1883 consensus_queue.cc:260] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.631604 1883 ts_tablet_manager.cc:1434] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20260501 13:57:39.591336 1608 tablet_service.cc:1558] Processing DeleteTablet for tablet 68cb7bd1a5aa4419b9c5176a287fe885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.637966 1124 master_service.cc:954] ReplaceTablet: received request to replace tablet 920d7f9748ae403d90573d612feeba59 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:39.642529 2093 ts_tablet_manager.cc:1916] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:39.645337 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4fe27772a87d4e42840a3ca5f2aed6d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:39.645581 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7c25358c17d94467828a54e6fd12be5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:39.646802 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fe27772a87d4e42840a3ca5f2aed6d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:39.647051 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c25358c17d94467828a54e6fd12be5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:39.636044 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee580bb9d3843f9bbeae8eef2c347ad" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:39.650377 1624 raft_consensus.cc:2468] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:39.651301 1130 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet c33a1f423bdb44f5843daffe8f73e74a (table default.Table_ecd46a159d2448f9ba09fc99e14a22a3 [id=d98d5437c5bc4e44bcfe27983dfa36e6]). Expected version 1 got 0
W20260501 13:57:39.658776 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 78658us
I20260501 13:57:39.660708 1421 leader_election.cc:304] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:39.660835 1608 tablet_service.cc:1558] Processing DeleteTablet for tablet 1c630c25ec5c46539936ec545ff4d292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.661222 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.662106 2320 raft_consensus.cc:2804] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:39.662020 2323 tablet.cc:1724] T d2c2873572de4889a16fda8fe31ad594 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260501 13:57:39.662439 2320 raft_consensus.cc:493] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:39.662744 2320 raft_consensus.cc:3060] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:39.665733 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 8d2f196f44be4fbc8059901380b71271 (table default.Table_360decd489b74f4b8ee93b008e57c45b [id=8cff6f6a2ebc447c979a45c9bfa438dc]) successfully deleted
I20260501 13:57:39.669143 2320 raft_consensus.cc:515] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.514128 2274 consensus_queue.cc:237] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.672595 2320 leader_election.cc:290] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:39.675817 1418 leader_election.cc:343] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 2ee580bb9d3843f9bbeae8eef2c347ad
I20260501 13:57:39.676306 1883 tablet_bootstrap.cc:492] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:39.679764 1899 tablet_replica.cc:333] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:39.680968 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet b507a4d4c9c54bc68d92d9286794a264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.681542 1899 raft_consensus.cc:2243] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:39.682240 1899 raft_consensus.cc:2272] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:39.691792 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee580bb9d3843f9bbeae8eef2c347ad" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:39.692168 1882 tablet_bootstrap.cc:654] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:39.692324 1624 raft_consensus.cc:3060] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:39.713686 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38c3eb388d474d718554775d65f58d78 consensus state has no leader
I20260501 13:57:39.716332 1123 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:39.718192 1624 raft_consensus.cc:2468] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:39.718750 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 920d7f9748ae403d90573d612feeba59
I20260501 13:57:39.719540 1899 ts_tablet_manager.cc:1916] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:39.719731 1421 leader_election.cc:304] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:39.720446 2320 raft_consensus.cc:2804] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:39.721688 1124 catalog_manager.cc:6741] ReplaceTablet: tablet 920d7f9748ae403d90573d612feeba59 deleted and replaced by tablet c07b1d2c40b74a93866db5124d5ee47f
I20260501 13:57:39.721763 2320 raft_consensus.cc:697] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:39.721863 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 920d7f9748ae403d90573d612feeba59 with delete_type TABLET_DATA_DELETED (replaced by tablet c07b1d2c40b74a93866db5124d5ee47f) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.723202 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 920d7f9748ae403d90573d612feeba59 with delete_type TABLET_DATA_DELETED (replaced by tablet c07b1d2c40b74a93866db5124d5ee47f) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.723800 2320 consensus_queue.cc:237] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.730732 1755 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d2f196f44be4fbc8059901380b71271 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.732800 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd02738e5f4ecfbfa06d5541614ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.733424 1755 tablet_service.cc:1558] Processing DeleteTablet for tablet 67b0675398d04be08bd29638b4ba2083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.734005 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.734599 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 920d7f9748ae403d90573d612feeba59 with delete_type TABLET_DATA_DELETED (replaced by tablet c07b1d2c40b74a93866db5124d5ee47f) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.745330 2302 consensus_queue.cc:1048] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:39.754928 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_f12194a7039e403eb237837a2e2653e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:39.756824 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f12194a7039e403eb237837a2e2653e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:39.762874 1883 tablet_bootstrap.cc:654] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:39.764417 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_3689c91dc5a941d0a884485d780a9ef4" } new_table_name: "default.Table_f1f44ff99c004e819d77e7d1e6580cb2" modify_external_catalogs: true
I20260501 13:57:39.773227 2282 raft_consensus.cc:493] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.773730 2282 raft_consensus.cc:515] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.776281 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d7a80ad4c4455a9ddee7a955e32635" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:39.777809 2282 leader_election.cc:290] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
W20260501 13:57:39.778661 1719 leader_election.cc:343] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.779690 2302 raft_consensus.cc:493] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.780726 2302 raft_consensus.cc:515] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.785331 2282 raft_consensus.cc:493] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.785852 2282 raft_consensus.cc:515] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.788731 2282 leader_election.cc:290] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:39.790614 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe89d656956440e8d0c1de0b01ce486" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:39.791838 1717 leader_election.cc:343] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.792773 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d7a80ad4c4455a9ddee7a955e32635" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:39.798408 1127 master_service.cc:954] ReplaceTablet: received request to replace tablet 104708eb7fc0412dad0bd7e14eee9d13 from {username='slave'} at 127.0.0.1:41700
W20260501 13:57:39.798619 1717 leader_election.cc:343] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.798965 1717 leader_election.cc:304] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:39.800055 2282 raft_consensus.cc:2749] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:39.804793 1883 tablet_bootstrap.cc:492] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:39.805204 1883 ts_tablet_manager.cc:1403] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.129s user 0.010s sys 0.000s
I20260501 13:57:39.789477 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe89d656956440e8d0c1de0b01ce486" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:39.806876 1719 leader_election.cc:343] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.807264 1719 leader_election.cc:304] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:39.807205 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_7b8be07c742f48d0af3422dfea143dd1" } modify_external_catalogs: true
I20260501 13:57:39.807832 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_7b8be07c742f48d0af3422dfea143dd1" } modify_external_catalogs: true
I20260501 13:57:39.808323 2282 raft_consensus.cc:2749] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:39.809442 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1409fbe5846a4bb099f31fb2d2e41382" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:39.817001 1484 raft_consensus.cc:2468] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:39.819141 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1409fbe5846a4bb099f31fb2d2e41382" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:39.819741 1782 raft_consensus.cc:2468] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:39.820132 2302 leader_election.cc:290] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:39.820920 1563 leader_election.cc:304] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:39.823635 2274 raft_consensus.cc:2804] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:39.824023 2274 raft_consensus.cc:493] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:39.824383 2274 raft_consensus.cc:3060] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:39.824882 1883 raft_consensus.cc:359] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.825505 1883 raft_consensus.cc:385] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.825767 1883 raft_consensus.cc:740] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.828958 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet e0356b19f0aa4197a41cc66d4440bffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.833861 1881 tablet_bootstrap.cc:654] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:39.833767 1124 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_536f8eeddbae410cbc4414090433de66" } modify_external_catalogs: true
I20260501 13:57:39.834278 1124 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_536f8eeddbae410cbc4414090433de66" } modify_external_catalogs: true
I20260501 13:57:39.835886 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet d361a2d82ecd456fb43f5e7fe3c83094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:39.838086 1096 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 48459.9us
I20260501 13:57:39.848531 1129 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet c33a1f423bdb44f5843daffe8f73e74a (table default.Table_ecd46a159d2448f9ba09fc99e14a22a3 [id=d98d5437c5bc4e44bcfe27983dfa36e6]). Expected version 1 got 0
I20260501 13:57:39.852190 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet e1d7a80ad4c4455a9ddee7a955e32635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.854491 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 with delete_type TABLET_DATA_DELETED (replaced by tablet f28db04617a449af8d87a554fb63e1e4) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.866014 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 6650472ec4a449c48c7957a9bbec88e3 (table default.Table_0d92e7b95c884bd5970e291f4189b0d9 [id=774a7eb90ba449b6aef225d819678439]). Expected version 1 got 0
I20260501 13:57:39.866882 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 9018d70c734a429aa59565ad67a46acc (table default.Table_a7b6759170be4cce860b79126e7bbc29 [id=454925118d6943458e68bb0089610a15]). Expected version 2 got 0
I20260501 13:57:39.872355 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet b98187c7843e42b09d55c851117c4ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:39.877300 1882 tablet_bootstrap.cc:492] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:39.877825 1882 ts_tablet_manager.cc:1403] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.255s user 0.010s sys 0.000s
I20260501 13:57:39.880285 1882 raft_consensus.cc:359] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.883325 1882 raft_consensus.cc:385] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.883661 1882 raft_consensus.cc:740] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.874958 2274 raft_consensus.cc:515] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.886864 1882 consensus_queue.cc:260] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.888752 1882 ts_tablet_manager.cc:1434] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.011s user 0.001s sys 0.004s
I20260501 13:57:39.888988 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1409fbe5846a4bb099f31fb2d2e41382" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:39.891000 1484 raft_consensus.cc:3060] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:39.892436 1882 tablet_bootstrap.cc:492] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:39.899363 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet c33a1f423bdb44f5843daffe8f73e74a (table default.Table_ecd46a159d2448f9ba09fc99e14a22a3 [id=d98d5437c5bc4e44bcfe27983dfa36e6]). Expected version 1 got 0
I20260501 13:57:39.906221 1882 tablet_bootstrap.cc:654] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:39.858433 1899 ts_tablet_manager.cc:1929] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260501 13:57:39.918954 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cfce57b777094ff0a135bd4f92d5f7dc consensus state has no leader
I20260501 13:57:39.919019 1899 log.cc:1199] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/ec4bd4cb1e70472aa09fc93afc3b1501
I20260501 13:57:39.919651 1899 ts_tablet_manager.cc:1950] T ec4bd4cb1e70472aa09fc93afc3b1501 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:39.891629 2282 raft_consensus.cc:493] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.903409 1883 consensus_queue.cc:260] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.920644 2282 raft_consensus.cc:515] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.923607 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d361a2d82ecd456fb43f5e7fe3c83094" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:39.923931 1882 tablet_bootstrap.cc:492] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:39.924333 1882 ts_tablet_manager.cc:1403] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.001s
W20260501 13:57:39.924860 1717 leader_election.cc:343] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.922426 2282 leader_election.cc:290] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:39.926728 1882 raft_consensus.cc:359] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.927299 1882 raft_consensus.cc:385] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.927531 1882 raft_consensus.cc:740] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.928045 1882 consensus_queue.cc:260] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.928881 2327 raft_consensus.cc:493] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.929283 2327 raft_consensus.cc:515] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.929903 1882 ts_tablet_manager.cc:1434] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260501 13:57:39.931092 2327 leader_election.cc:290] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:39.932906 1130 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_d986c7fb196a4dc2a25b8bba537072c9 [id=b8901f06222746b8bf418391866b92cc] alter complete (version 4)
I20260501 13:57:39.932910 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6650472ec4a449c48c7957a9bbec88e3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:39.933912 1882 tablet_bootstrap.cc:492] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:39.934063 1780 raft_consensus.cc:2468] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:39.935683 1563 leader_election.cc:304] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:39.936609 1883 ts_tablet_manager.cc:1434] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.131s user 0.003s sys 0.006s
I20260501 13:57:39.937052 2328 raft_consensus.cc:2804] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:39.937373 2328 raft_consensus.cc:493] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:39.937668 2328 raft_consensus.cc:3060] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:39.941373 1881 tablet_bootstrap.cc:492] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:39.941769 1881 ts_tablet_manager.cc:1403] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.361s user 0.008s sys 0.000s
I20260501 13:57:39.945047 1127 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 51ba7b8a7f4d4ec6ba4af775596d3b59
W20260501 13:57:39.947170 1563 consensus_peers.cc:597] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Couldn't send request to peer 0ed1f9a545474e76a661c32bf11a7fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c789ca9e3c748d5bc79f4e1b0ea8fd1. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:39.947921 1882 tablet_bootstrap.cc:654] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:39.947904 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ec911a9efe2c4a70a032be2376620b96" } new_table_name: "default.Table_fe9befde21164496ad12d593c8e08660" modify_external_catalogs: true
I20260501 13:57:39.948244 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.949417 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6650472ec4a449c48c7957a9bbec88e3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:39.951085 1564 leader_election.cc:343] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.954265 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:39.955686 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d361a2d82ecd456fb43f5e7fe3c83094" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:39.956827 2327 raft_consensus.cc:493] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:39.957239 2327 raft_consensus.cc:515] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.957917 1127 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet c33a1f423bdb44f5843daffe8f73e74a (table default.Table_ecd46a159d2448f9ba09fc99e14a22a3 [id=d98d5437c5bc4e44bcfe27983dfa36e6]). Expected version 1 got 0
W20260501 13:57:39.959569 1719 leader_election.cc:343] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.959982 1719 leader_election.cc:304] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:39.963089 2282 raft_consensus.cc:2749] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:39.963627 2282 consensus_queue.cc:1048] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:39.966233 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe89d656956440e8d0c1de0b01ce486" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:39.966931 1780 raft_consensus.cc:2468] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:39.968040 1563 leader_election.cc:304] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:39.968866 2302 raft_consensus.cc:2804] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:39.969195 2302 raft_consensus.cc:493] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:39.969558 2302 raft_consensus.cc:3060] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:39.959429 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet b125d1a190f947a5b2ef942f04408c51
I20260501 13:57:39.972889 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet b125d1a190f947a5b2ef942f04408c51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:39.974779 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_645fbdfab04b4def8a0d842bd01c2ade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:39.975546 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe89d656956440e8d0c1de0b01ce486" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:39.976230 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_645fbdfab04b4def8a0d842bd01c2ade in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:39.976997 1564 leader_election.cc:343] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:39.962033 1881 raft_consensus.cc:359] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.977506 2327 leader_election.cc:290] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:39.977686 1881 raft_consensus.cc:385] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:39.977952 1881 raft_consensus.cc:740] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:39.978641 1881 consensus_queue.cc:260] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:39.983775 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7f0f65b600aa434cbe776589a4ec2855"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:39.984138 2328 raft_consensus.cc:515] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:39.990332 1117 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 3b0ed592fd3642c194bb946c8c6003b4 (table default.Table_407a6cfd065142f4a2debee031077ab9 [id=a5184058a20f431280f6ccd10a290f27], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3b0ed592fd3642c194bb946c8c6003b4 consensus state has no leader
I20260501 13:57:39.996399 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:40.002069 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet b125d1a190f947a5b2ef942f04408c51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:40.005307 1899 tablet_replica.cc:333] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:40.006042 1899 raft_consensus.cc:2243] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:40.006453 1899 raft_consensus.cc:2272] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:40.009431 1899 ts_tablet_manager.cc:1916] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:40.011715 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet ec4bd4cb1e70472aa09fc93afc3b1501 (table default.Table_6ee51a9fc8ae4340b14c506cf9c450c2 [id=e61fe8a70f66497db17db0afacc35100]) successfully deleted
I20260501 13:57:40.007611 1883 tablet_bootstrap.cc:492] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:40.021258 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f0f65b600aa434cbe776589a4ec2855 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:39.993826 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_24c07f00f29c4f20835097d84de6e17a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:40.024847 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24c07f00f29c4f20835097d84de6e17a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.025259 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6650472ec4a449c48c7957a9bbec88e3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
W20260501 13:57:40.026631 1564 leader_election.cc:343] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.014621 2327 raft_consensus.cc:493] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.027237 2327 raft_consensus.cc:515] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.029707 1931 raft_consensus.cc:493] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.030166 1931 raft_consensus.cc:515] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:40.031046 1563 leader_election.cc:343] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: b98187c7843e42b09d55c851117c4ac8
I20260501 13:57:40.031976 1931 leader_election.cc:290] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.033510 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "920d7f9748ae403d90573d612feeba59" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:40.035071 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "920d7f9748ae403d90573d612feeba59" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:40.037272 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b98187c7843e42b09d55c851117c4ac8" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:40.038693 1564 leader_election.cc:343] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.039139 1564 leader_election.cc:304] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:40.039526 1418 leader_election.cc:304] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:40.040985 2218 raft_consensus.cc:2749] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:40.042279 2327 leader_election.cc:290] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.042843 2327 raft_consensus.cc:2749] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:40.045081 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1409fbe5846a4bb099f31fb2d2e41382" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:40.045712 1780 raft_consensus.cc:3060] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:40.047118 2047 ts_tablet_manager.cc:1929] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:40.047551 2047 log.cc:1199] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/2ff05404061447199d936d005afda653
I20260501 13:57:40.048243 2047 ts_tablet_manager.cc:1950] T 2ff05404061447199d936d005afda653 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:40.049981 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 2ff05404061447199d936d005afda653 (table default.Table_26541c8dd6a347a78e26e6cb55fa3b06 [id=15dd9506496343159cc3f4de3afcef5e]) successfully deleted
I20260501 13:57:40.050880 1483 raft_consensus.cc:3060] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:40.051998 1883 tablet_bootstrap.cc:654] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.053057 1882 tablet_bootstrap.cc:492] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:40.053483 1882 ts_tablet_manager.cc:1403] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.120s user 0.026s sys 0.019s
I20260501 13:57:40.056979 1882 raft_consensus.cc:359] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.057601 1882 raft_consensus.cc:385] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.057848 1882 raft_consensus.cc:740] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:40.058403 1882 consensus_queue.cc:260] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.060388 1882 ts_tablet_manager.cc:1434] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260501 13:57:40.061434 1882 tablet_bootstrap.cc:492] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:40.063808 2274 leader_election.cc:290] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.067169 1882 tablet_bootstrap.cc:654] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.073630 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6650472ec4a449c48c7957a9bbec88e3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:40.074245 1782 raft_consensus.cc:3060] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:40.051120 2047 tablet_replica.cc:333] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:40.058408 1881 ts_tablet_manager.cc:1434] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.116s user 0.006s sys 0.002s
I20260501 13:57:40.076056 1881 tablet_bootstrap.cc:492] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:40.067699 1484 raft_consensus.cc:2468] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:40.083573 2047 raft_consensus.cc:2243] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
W20260501 13:57:40.083906 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd633c026195429ca69d8041809bee45 has no consensus state
I20260501 13:57:40.084432 2047 pending_rounds.cc:70] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: Trying to abort 1 pending ops.
I20260501 13:57:40.084673 2047 pending_rounds.cc:77] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229241568264192 op_type: NO_OP noop_request { }
I20260501 13:57:40.084990 2047 raft_consensus.cc:2889] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:40.085301 2047 raft_consensus.cc:2272] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:40.088925 1883 tablet_bootstrap.cc:492] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:40.089334 1883 ts_tablet_manager.cc:1403] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.001s
I20260501 13:57:40.089499 2339 raft_consensus.cc:493] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.090008 2339 raft_consensus.cc:515] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.091696 1883 raft_consensus.cc:359] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.092068 1882 tablet_bootstrap.cc:492] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:40.092283 1883 raft_consensus.cc:385] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.092453 1882 ts_tablet_manager.cc:1403] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20260501 13:57:40.092533 1883 raft_consensus.cc:740] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:40.092727 2339 leader_election.cc:290] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:40.094252 1782 raft_consensus.cc:2468] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:40.095021 1882 raft_consensus.cc:359] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.095638 1882 raft_consensus.cc:385] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.095526 1563 leader_election.cc:304] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:40.095929 1882 raft_consensus.cc:740] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:40.096606 1882 consensus_queue.cc:260] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.097787 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94abe840b1994c0cb1f7ac7f4ac09cd9" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:40.099330 1881 tablet_bootstrap.cc:654] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.099468 2328 leader_election.cc:290] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.103168 2327 raft_consensus.cc:2804] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:40.103674 2327 raft_consensus.cc:697] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:40.104449 2327 consensus_queue.cc:237] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:40.109510 1119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.188:38815, user_credentials={real_user=slave}} blocked reactor thread for 46106.7us
I20260501 13:57:40.111905 1882 ts_tablet_manager.cc:1434] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
W20260501 13:57:40.112263 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 73624.7us
I20260501 13:57:40.112783 1882 tablet_bootstrap.cc:492] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
W20260501 13:57:40.121114 1717 leader_election.cc:343] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.018514 2302 raft_consensus.cc:515] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.126353 2302 leader_election.cc:290] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.127180 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe89d656956440e8d0c1de0b01ce486" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:40.127718 1782 raft_consensus.cc:3060] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:40.135401 1882 tablet_bootstrap.cc:654] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.137099 1780 raft_consensus.cc:2468] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:40.138521 1563 leader_election.cc:304] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:40.139689 2093 ts_tablet_manager.cc:1929] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:40.140187 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_b4da3389e7934799aa303ed608a26082" } new_table_name: "default.Table_f845f5c6881e4f2587c0decd568ffabc" modify_external_catalogs: true
I20260501 13:57:40.140401 2344 raft_consensus.cc:2804] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:40.140511 2047 ts_tablet_manager.cc:1916] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:40.140856 2344 raft_consensus.cc:697] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:40.141672 2344 consensus_queue.cc:237] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.142980 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94abe840b1994c0cb1f7ac7f4ac09cd9" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:40.143751 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_e3ff5a5e561a442d9af7f6c8317bd75b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:40.144310 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:40.144579 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet b125d1a190f947a5b2ef942f04408c51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:40.146870 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe89d656956440e8d0c1de0b01ce486" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:40.148274 1484 raft_consensus.cc:3060] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:40.150758 1883 consensus_queue.cc:260] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.152729 1883 ts_tablet_manager.cc:1434] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.063s user 0.006s sys 0.000s
I20260501 13:57:40.153486 2282 consensus_queue.cc:1048] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:40.156716 1126 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9944c024418466d997d764eb2f3ad50 has no consensus state
I20260501 13:57:40.158869 1882 tablet_bootstrap.cc:492] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:40.159292 1882 ts_tablet_manager.cc:1403] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.003s
I20260501 13:57:40.159973 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_56892e9ecd4b4b0b9133bc936b37c1c4" } modify_external_catalogs: true
I20260501 13:57:40.163045 1899 ts_tablet_manager.cc:1929] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:40.163175 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_56892e9ecd4b4b0b9133bc936b37c1c4" } modify_external_catalogs: true
I20260501 13:57:40.163610 1899 log.cc:1199] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/0669a229d6d54eacac9889ad18e06f30
I20260501 13:57:40.164230 1899 ts_tablet_manager.cc:1950] T 0669a229d6d54eacac9889ad18e06f30 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:40.164002 1882 raft_consensus.cc:359] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.164602 1882 raft_consensus.cc:385] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.164848 1882 raft_consensus.cc:740] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
W20260501 13:57:40.172133 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3ff5a5e561a442d9af7f6c8317bd75b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.165398 1882 consensus_queue.cc:260] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.173930 1882 ts_tablet_manager.cc:1434] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260501 13:57:40.177352 1130 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_a27ca155bb494a99bc23aefa4e6823f2" } modify_external_catalogs: true
I20260501 13:57:40.178119 1882 tablet_bootstrap.cc:492] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:40.177956 1130 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_a27ca155bb494a99bc23aefa4e6823f2" } modify_external_catalogs: true
I20260501 13:57:40.181468 1883 tablet_bootstrap.cc:492] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:40.186930 1564 leader_election.cc:343] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:40.187602 1719 leader_election.cc:343] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.187985 1719 leader_election.cc:304] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:40.188614 2339 raft_consensus.cc:2749] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:40.190973 1882 tablet_bootstrap.cc:654] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.209281 1899 tablet_replica.cc:333] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:40.211606 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 0669a229d6d54eacac9889ad18e06f30 (table default.Table_641d4d711c4d49319daa2cb0cf2714c6 [id=535a372705e4460ab657255e3f1be962]) successfully deleted
W20260501 13:57:40.213894 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 68cb7bd1a5aa4419b9c5176a287fe885 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68cb7bd1a5aa4419b9c5176a287fe885 already in progress: creating tablet
W20260501 13:57:40.215006 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 1c630c25ec5c46539936ec545ff4d292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c630c25ec5c46539936ec545ff4d292 already in progress: creating tablet
I20260501 13:57:40.218425 1882 tablet_bootstrap.cc:492] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:40.218828 1882 ts_tablet_manager.cc:1403] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.002s
I20260501 13:57:40.220072 1881 tablet_bootstrap.cc:492] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:40.220495 1881 ts_tablet_manager.cc:1403] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.145s user 0.006s sys 0.004s
I20260501 13:57:40.221187 1882 raft_consensus.cc:359] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.221805 1882 raft_consensus.cc:385] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.222043 1882 raft_consensus.cc:740] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:40.222620 1882 consensus_queue.cc:260] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.230110 1882 ts_tablet_manager.cc:1434] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20260501 13:57:40.230978 1882 tablet_bootstrap.cc:492] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:40.140026 2093 log.cc:1199] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/2ff05404061447199d936d005afda653
I20260501 13:57:40.243095 2093 ts_tablet_manager.cc:1950] T 2ff05404061447199d936d005afda653 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:40.246151 1881 raft_consensus.cc:359] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.246837 1881 raft_consensus.cc:385] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.247185 1881 raft_consensus.cc:740] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:40.247852 1881 consensus_queue.cc:260] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.258476 1899 raft_consensus.cc:2243] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:40.246531 1882 tablet_bootstrap.cc:654] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.270751 2327 raft_consensus.cc:493] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.271222 2327 raft_consensus.cc:515] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.273095 2327 leader_election.cc:290] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.276738 1881 ts_tablet_manager.cc:1434] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.056s user 0.006s sys 0.000s
I20260501 13:57:40.280939 2327 raft_consensus.cc:493] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.281374 2327 raft_consensus.cc:515] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.282948 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9c92846d4e4255a1c8ce3835a0a18d" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:40.283830 2327 leader_election.cc:290] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.284376 1131 master_service.cc:954] ReplaceTablet: received request to replace tablet 1bb7fdcad3a54317b43bfd58f08e89d0 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:40.285049 1881 tablet_bootstrap.cc:492] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:40.286988 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 188) took 1267 ms. Trace:
I20260501 13:57:40.287284 2093 rpcz_store.cc:276] 0501 13:57:39.019652 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.041628 (+ 21976us) service_pool.cc:224] Handling call
0501 13:57:39.513978 (+472350us) ts_tablet_manager.cc:1151] Deleting tablet 2ff05404061447199d936d005afda653
0501 13:57:39.514022 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:39.875780 (+361758us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.139577 (+263797us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.286927 (+147350us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14066,"tablet-delete.queue_time_us":471828}
I20260501 13:57:40.288368 2093 tablet_replica.cc:333] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:40.289049 2093 raft_consensus.cc:2243] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:40.289439 2093 raft_consensus.cc:2272] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:40.289215 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_de443751750240b6a298d8aca41e6b56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:40.291049 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_443326309dee4e12b1b00584159d6551" } new_table_name: "default.Table_cbe7b051f36648d395177efd3e101e2d" modify_external_catalogs: true
I20260501 13:57:40.297411 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9c92846d4e4255a1c8ce3835a0a18d" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
W20260501 13:57:40.298671 1563 leader_election.cc:343] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 1ba63c153ef04db891cae46a71439455
W20260501 13:57:40.295349 1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 96932.7us
I20260501 13:57:40.306005 1780 raft_consensus.cc:2468] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:40.307166 1563 leader_election.cc:304] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:40.307488 1883 tablet_bootstrap.cc:654] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.308017 2327 raft_consensus.cc:2804] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:40.308389 2327 raft_consensus.cc:493] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:40.308694 2327 raft_consensus.cc:3060] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:40.313045 1899 raft_consensus.cc:2272] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:40.314373 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 967658b1be204c638fc34983293ef5ab consensus state has no leader
I20260501 13:57:40.316260 2327 raft_consensus.cc:515] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.295311 2282 raft_consensus.cc:493] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.317229 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 2ff05404061447199d936d005afda653 (table default.Table_26541c8dd6a347a78e26e6cb55fa3b06 [id=15dd9506496343159cc3f4de3afcef5e]) successfully deleted
W20260501 13:57:40.317358 1564 leader_election.cc:343] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.317754 2282 raft_consensus.cc:515] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.319470 1782 raft_consensus.cc:2468] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
W20260501 13:57:40.319537 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de443751750240b6a298d8aca41e6b56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.320796 1563 leader_election.cc:304] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:40.321559 2282 leader_election.cc:290] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:40.321595 2327 leader_election.cc:290] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.321803 2302 raft_consensus.cc:2804] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:40.322839 2302 raft_consensus.cc:697] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:40.323980 2302 consensus_queue.cc:237] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.322324 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9c92846d4e4255a1c8ce3835a0a18d" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:40.325688 1780 raft_consensus.cc:3060] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:40.328797 1899 ts_tablet_manager.cc:1916] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:40.331362 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38c3eb388d474d718554775d65f58d78" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:40.339602 1780 raft_consensus.cc:2468] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:40.340948 1882 tablet_bootstrap.cc:492] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:40.341358 1882 ts_tablet_manager.cc:1403] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.111s user 0.006s sys 0.012s
I20260501 13:57:40.342149 1563 leader_election.cc:304] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:40.343098 2327 raft_consensus.cc:2804] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:40.343513 2327 raft_consensus.cc:697] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:40.344219 2327 consensus_queue.cc:237] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.345501 1882 raft_consensus.cc:359] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:40.345966 1719 leader_election.cc:343] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.346081 1882 raft_consensus.cc:385] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.346319 1882 raft_consensus.cc:740] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:40.351176 1882 consensus_queue.cc:260] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.353103 1882 ts_tablet_manager.cc:1434] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260501 13:57:40.353904 1882 tablet_bootstrap.cc:492] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:40.356204 2047 ts_tablet_manager.cc:1929] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:40.356704 2047 log.cc:1199] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/e41b43329e054442bc8697cc28161954
I20260501 13:57:40.357443 2047 ts_tablet_manager.cc:1950] T e41b43329e054442bc8697cc28161954 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:40.358237 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 196) took 1074 ms. Trace:
I20260501 13:57:40.361850 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_dc19bcebd0934f20b6dc5a1da2f67f21" } new_table_name: "default.Table_8f4a53cf4cf54f5e93c7bb1ca296d4cd" modify_external_catalogs: true
I20260501 13:57:40.362120 2093 ts_tablet_manager.cc:1916] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:40.366881 1883 tablet_bootstrap.cc:492] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:40.367449 1883 ts_tablet_manager.cc:1403] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.186s user 0.010s sys 0.003s
I20260501 13:57:40.369635 2327 raft_consensus.cc:493] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.370108 2327 raft_consensus.cc:515] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.373395 2327 leader_election.cc:290] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:40.374392 1563 leader_election.cc:343] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 90a57bc58e854e469897a04b2d434d06
I20260501 13:57:40.384604 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_5d773086180e4cd09927c8b2fe2e650b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:40.386083 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d773086180e4cd09927c8b2fe2e650b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.387197 2302 raft_consensus.cc:493] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.387634 2302 raft_consensus.cc:515] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.390450 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9018d70c734a429aa59565ad67a46acc" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:40.391048 1780 raft_consensus.cc:2468] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:40.392227 1563 leader_election.cc:304] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:40.393379 2327 raft_consensus.cc:2804] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:40.393690 2327 raft_consensus.cc:493] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:40.393976 2327 raft_consensus.cc:3060] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:40.399428 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81daa397384d4afb816071ee665a1dcd consensus state has no leader
I20260501 13:57:40.398901 2302 leader_election.cc:290] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.402362 2327 raft_consensus.cc:515] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.408879 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9018d70c734a429aa59565ad67a46acc" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:40.409454 1780 raft_consensus.cc:3060] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:40.416265 1780 raft_consensus.cc:2468] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:40.417434 1563 leader_election.cc:304] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:40.418318 2302 raft_consensus.cc:2804] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:40.418599 2047 rpcz_store.cc:276] 0501 13:57:39.284142 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.306811 (+ 22669us) service_pool.cc:224] Handling call
0501 13:57:40.050983 (+744172us) ts_tablet_manager.cc:1151] Deleting tablet e41b43329e054442bc8697cc28161954
0501 13:57:40.051021 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:40.200622 (+149601us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.356057 (+155435us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.358163 (+ 2106us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8241,"tablet-delete.queue_time_us":743193}
I20260501 13:57:40.418855 2302 raft_consensus.cc:697] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:40.420050 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet e41b43329e054442bc8697cc28161954 (table default.Table_2edf812ced814de8acadfe87bcad5893 [id=694ff2769ea04445811ca8d8a92854fe]) successfully deleted
I20260501 13:57:40.420918 2047 tablet_replica.cc:333] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:40.421684 2047 raft_consensus.cc:2243] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:40.421314 2302 consensus_queue.cc:237] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.422266 2047 raft_consensus.cc:2272] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:40.424234 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90a57bc58e854e469897a04b2d434d06" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:40.424628 2327 leader_election.cc:290] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.398663 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9c92846d4e4255a1c8ce3835a0a18d" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:40.421710 2093 ts_tablet_manager.cc:1929] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:40.428462 2093 log.cc:1199] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/2ee580bb9d3843f9bbeae8eef2c347ad
I20260501 13:57:40.429121 2093 ts_tablet_manager.cc:1950] T 2ee580bb9d3843f9bbeae8eef2c347ad P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:40.422868 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ba63c153ef04db891cae46a71439455" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:40.430599 1564 leader_election.cc:343] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.430565 1481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9018d70c734a429aa59565ad67a46acc" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:40.430974 1564 leader_election.cc:304] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:40.432781 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9018d70c734a429aa59565ad67a46acc" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:40.433388 1564 leader_election.cc:343] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.433728 1564 leader_election.cc:304] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:40.433617 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_bcf66326bed34b2ba3bba85f08f2573b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:40.434734 2344 raft_consensus.cc:2749] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:40.435261 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcf66326bed34b2ba3bba85f08f2573b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.435284 2344 raft_consensus.cc:2749] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:40.437114 1564 leader_election.cc:343] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.437067 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38c3eb388d474d718554775d65f58d78" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:40.437911 1564 leader_election.cc:343] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:40.438339 1717 leader_election.cc:343] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.438743 1717 leader_election.cc:304] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:40.438810 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 189) took 1396 ms. Trace:
I20260501 13:57:40.439384 2282 raft_consensus.cc:2749] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:40.439456 2327 raft_consensus.cc:493] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.440145 2327 raft_consensus.cc:515] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:40.443939 1563 leader_election.cc:343] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 863a10cfd14c4694b7c6141c0851167a
I20260501 13:57:40.444550 2327 leader_election.cc:290] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.446223 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "863a10cfd14c4694b7c6141c0851167a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:40.446900 1485 raft_consensus.cc:2468] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:40.439164 2093 rpcz_store.cc:276] 0501 13:57:39.042499 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.042834 (+ 335us) service_pool.cc:224] Handling call
0501 13:57:40.288239 (+1245405us) ts_tablet_manager.cc:1151] Deleting tablet 2ee580bb9d3843f9bbeae8eef2c347ad
0501 13:57:40.288276 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:40.393960 (+105684us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.421597 (+ 27637us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.438742 (+ 17145us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1244951}
W20260501 13:57:40.450744 1564 leader_election.cc:343] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.451320 2093 tablet_replica.cc:333] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:40.451905 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 2ee580bb9d3843f9bbeae8eef2c347ad (table default.Table_abb70b9287914fbc97d3dcf78ac373bf [id=fe9f5542710e48e8bbf9a48e07c412d6]) successfully deleted
I20260501 13:57:40.452028 2093 raft_consensus.cc:2243] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:40.452505 2093 raft_consensus.cc:2272] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:40.454222 2093 ts_tablet_manager.cc:1916] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:40.454166 1564 leader_election.cc:304] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:40.455173 2327 raft_consensus.cc:2804] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:40.455492 2327 raft_consensus.cc:493] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:40.455794 2327 raft_consensus.cc:3060] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:40.369794 1883 raft_consensus.cc:359] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.461218 1883 raft_consensus.cc:385] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.461480 1883 raft_consensus.cc:740] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:40.462044 1883 consensus_queue.cc:260] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.464311 1883 ts_tablet_manager.cc:1434] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.097s user 0.004s sys 0.000s
I20260501 13:57:40.473979 1124 master_service.cc:954] ReplaceTablet: received request to replace tablet b8d0f6d1f38b428291dba22015ac6959 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:40.481961 1931 raft_consensus.cc:493] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.482550 1931 raft_consensus.cc:515] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.485725 1931 leader_election.cc:290] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.474752 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_58189c0095284474a78cf7aa10654434"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:40.487840 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58189c0095284474a78cf7aa10654434 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:40.489068 1096 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 61891.5us
I20260501 13:57:40.494024 1883 tablet_bootstrap.cc:492] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:40.485314 2047 ts_tablet_manager.cc:1916] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:40.499332 1418 leader_election.cc:343] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 9c789ca9e3c748d5bc79f4e1b0ea8fd1
W20260501 13:57:40.503839 1418 consensus_peers.cc:597] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 -> Peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Couldn't send request to peer 0ed1f9a545474e76a661c32bf11a7fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ee580bb9d3843f9bbeae8eef2c347ad. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:40.509675 1881 tablet_bootstrap.cc:654] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.513059 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet b8d0f6d1f38b428291dba22015ac6959
I20260501 13:57:40.515497 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet b8d0f6d1f38b428291dba22015ac6959 with delete_type TABLET_DATA_DELETED (replaced by tablet ab08e04739cf42f3b7ae2e22eb5fbc36) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:40.517664 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c32278ab7552423e922a3248d158e499" } new_table_name: "default.Table_230433f32f184024be1ec4c52d7acea8" modify_external_catalogs: true
I20260501 13:57:40.523429 1124 catalog_manager.cc:6741] ReplaceTablet: tablet b8d0f6d1f38b428291dba22015ac6959 deleted and replaced by tablet ab08e04739cf42f3b7ae2e22eb5fbc36
I20260501 13:57:40.527926 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet b8d0f6d1f38b428291dba22015ac6959 with delete_type TABLET_DATA_DELETED (replaced by tablet ab08e04739cf42f3b7ae2e22eb5fbc36) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:40.477339 2327 raft_consensus.cc:515] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.483692 2093 ts_tablet_manager.cc:1929] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:57:40.486387 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c789ca9e3c748d5bc79f4e1b0ea8fd1" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:40.532596 2093 log.cc:1199] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/8d2f196f44be4fbc8059901380b71271
I20260501 13:57:40.491911 2344 consensus_queue.cc:1048] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:40.533293 2093 ts_tablet_manager.cc:1950] T 8d2f196f44be4fbc8059901380b71271 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
W20260501 13:57:40.501741 1563 consensus_peers.cc:597] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Couldn't send request to peer 0ed1f9a545474e76a661c32bf11a7fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b507a4d4c9c54bc68d92d9286794a264. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:40.535485 1421 leader_election.cc:304] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
W20260501 13:57:40.514638 2360 log.cc:927] Time spent T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf: Append to log took a long time: real 0.159s user 0.001s sys 0.000s
I20260501 13:57:40.529052 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet b8d0f6d1f38b428291dba22015ac6959 with delete_type TABLET_DATA_DELETED (replaced by tablet ab08e04739cf42f3b7ae2e22eb5fbc36) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:40.538105 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 199) took 1176 ms. Trace:
I20260501 13:57:40.538411 2093 rpcz_store.cc:276] 0501 13:57:39.361690 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.361806 (+ 116us) service_pool.cc:224] Handling call
0501 13:57:40.451177 (+1089371us) ts_tablet_manager.cc:1151] Deleting tablet 8d2f196f44be4fbc8059901380b71271
0501 13:57:40.451221 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:40.466116 (+ 14895us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.483565 (+ 17449us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.538049 (+ 54484us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1088851}
I20260501 13:57:40.540056 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 8d2f196f44be4fbc8059901380b71271 (table default.Table_360decd489b74f4b8ee93b008e57c45b [id=8cff6f6a2ebc447c979a45c9bfa438dc]) successfully deleted
I20260501 13:57:40.539963 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_96180f484b044c0b85bb2f5d8a14d194"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:40.541450 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96180f484b044c0b85bb2f5d8a14d194 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.542392 1931 raft_consensus.cc:2749] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:57:40.544476 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 202) took 1175 ms. Trace:
I20260501 13:57:40.544785 2093 rpcz_store.cc:276] 0501 13:57:39.374551 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.374719 (+ 168us) service_pool.cc:224] Handling call
0501 13:57:40.539240 (+1164521us) ts_tablet_manager.cc:1151] Deleting tablet 0669a229d6d54eacac9889ad18e06f30
0501 13:57:40.539269 (+ 29us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:40.539506 (+ 237us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164013}
I20260501 13:57:40.545882 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 198) took 1184 ms. Trace:
W20260501 13:57:40.545826 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 0669a229d6d54eacac9889ad18e06f30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0669a229d6d54eacac9889ad18e06f30 already in progress: creating tablet
I20260501 13:57:40.546159 2093 rpcz_store.cc:276] 0501 13:57:39.361417 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.401125 (+ 39708us) service_pool.cc:224] Handling call
0501 13:57:40.545570 (+1144445us) ts_tablet_manager.cc:1151] Deleting tablet 569a0240b1274d9bb830addbaf1ec343
0501 13:57:40.545610 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:40.545832 (+ 222us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085779}
I20260501 13:57:40.547951 2093 tablet_replica.cc:333] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
W20260501 13:57:40.548418 1563 leader_election.cc:343] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 863a10cfd14c4694b7c6141c0851167a
I20260501 13:57:40.549490 2327 leader_election.cc:290] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:40.549824 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 569a0240b1274d9bb830addbaf1ec343 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 569a0240b1274d9bb830addbaf1ec343 already in progress: creating tablet
W20260501 13:57:40.550529 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 68cb7bd1a5aa4419b9c5176a287fe885 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68cb7bd1a5aa4419b9c5176a287fe885 already in progress: creating tablet
I20260501 13:57:40.550494 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "863a10cfd14c4694b7c6141c0851167a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:40.551019 1485 raft_consensus.cc:3060] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:40.551115 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 1c630c25ec5c46539936ec545ff4d292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c630c25ec5c46539936ec545ff4d292 already in progress: creating tablet
I20260501 13:57:40.554433 1899 ts_tablet_manager.cc:1929] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:40.554817 1899 log.cc:1199] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/920d7f9748ae403d90573d612feeba59
I20260501 13:57:40.555487 1899 ts_tablet_manager.cc:1950] T 920d7f9748ae403d90573d612feeba59 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:40.558200 1485 raft_consensus.cc:2468] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:40.568994 2302 raft_consensus.cc:493] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 13:57:40.569406 1125 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00e64653da9847cbb1fab02e002993b5 has no consensus state
I20260501 13:57:40.569550 2302 raft_consensus.cc:515] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.570708 2093 raft_consensus.cc:2243] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:40.572146 2093 raft_consensus.cc:2272] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:40.535972 1882 tablet_bootstrap.cc:654] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.574931 1883 tablet_bootstrap.cc:654] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:40.586794 1563 leader_election.cc:343] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: eeae4369be954d9d9bad846484afb6ff
I20260501 13:57:40.588397 2302 leader_election.cc:290] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.575984 1881 tablet_bootstrap.cc:492] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:40.590276 1881 ts_tablet_manager.cc:1403] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.305s user 0.009s sys 0.000s
I20260501 13:57:40.593375 2093 ts_tablet_manager.cc:1916] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:40.594802 1881 raft_consensus.cc:359] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.595459 1881 raft_consensus.cc:385] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.595714 1881 raft_consensus.cc:740] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:40.596305 1881 consensus_queue.cc:260] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.601914 1882 tablet_bootstrap.cc:492] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:40.602391 1882 ts_tablet_manager.cc:1403] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.249s user 0.009s sys 0.002s
I20260501 13:57:40.602572 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_8f4a53cf4cf54f5e93c7bb1ca296d4cd" } new_table_name: "default.Table_ecd622b82fa9463198fe36afb1458343" modify_external_catalogs: true
I20260501 13:57:40.605019 1882 raft_consensus.cc:359] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.606756 2282 raft_consensus.cc:493] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.607203 2282 raft_consensus.cc:515] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.608906 2282 leader_election.cc:290] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:40.610767 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "406e1af8b7d9488f88697206daa87c05" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:40.612208 1719 leader_election.cc:343] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.607156 1882 raft_consensus.cc:385] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.613368 1899 tablet_replica.cc:333] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:40.614070 1899 raft_consensus.cc:2243] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:40.614502 1899 raft_consensus.cc:2272] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:40.616050 1899 ts_tablet_manager.cc:1916] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:40.619414 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81daa397384d4afb816071ee665a1dcd consensus state has no leader
I20260501 13:57:40.620535 1882 raft_consensus.cc:740] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:40.626184 1883 tablet_bootstrap.cc:492] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:40.626672 1883 ts_tablet_manager.cc:1403] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.133s user 0.009s sys 0.000s
I20260501 13:57:40.623008 1882 consensus_queue.cc:260] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.630333 2093 ts_tablet_manager.cc:1929] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:40.630746 2093 log.cc:1199] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/9c789ca9e3c748d5bc79f4e1b0ea8fd1
I20260501 13:57:40.632339 2093 ts_tablet_manager.cc:1950] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:40.612996 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 920d7f9748ae403d90573d612feeba59 (table default.Table_f0801819716e426f92844203c4a06bdc [id=068f5ac17d494d0b8639cbdc07a04597]) successfully deleted
W20260501 13:57:40.636083 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 8d2f196f44be4fbc8059901380b71271 because the tablet was not found. No further retry: Not found: Tablet not found: 8d2f196f44be4fbc8059901380b71271
I20260501 13:57:40.637323 1882 ts_tablet_manager.cc:1434] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20260501 13:57:40.641809 1882 tablet_bootstrap.cc:492] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:40.643684 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 (table default.Table_2a1ca2e987e243ad9f1c064a28866dd3 [id=ae0dad3b628b454d899e4912c8013d14]) successfully deleted
I20260501 13:57:40.653865 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_bd8182bfb41b4025b826d479090f8f09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:40.655570 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd8182bfb41b4025b826d479090f8f09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.666709 1883 raft_consensus.cc:359] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.667344 1883 raft_consensus.cc:385] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.667603 1883 raft_consensus.cc:740] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:40.668249 1883 consensus_queue.cc:260] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.673408 2282 raft_consensus.cc:493] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.673833 2282 raft_consensus.cc:515] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.674019 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_2c4ab34d8a9b4072b1cc2cb7c5ca3203"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:40.675483 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c4ab34d8a9b4072b1cc2cb7c5ca3203 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.676245 2282 leader_election.cc:290] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:40.679642 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c736959df334aee857dbaae0f7a1e1a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:40.682379 1717 leader_election.cc:343] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.643651 2093 tablet_replica.cc:333] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:40.704252 2093 raft_consensus.cc:2243] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:40.704991 2093 pending_rounds.cc:70] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: Trying to abort 1 pending ops.
I20260501 13:57:40.705219 2093 pending_rounds.cc:77] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229249272995840 op_type: NO_OP noop_request { }
I20260501 13:57:40.705538 2093 raft_consensus.cc:2889] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:40.705839 2093 raft_consensus.cc:2272] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:40.707031 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "406e1af8b7d9488f88697206daa87c05" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:40.683521 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c736959df334aee857dbaae0f7a1e1a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:40.711562 1719 leader_election.cc:343] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.711939 1719 leader_election.cc:304] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:40.713008 1717 leader_election.cc:343] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.713369 1717 leader_election.cc:304] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:40.714210 2354 raft_consensus.cc:2749] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:40.714713 2354 raft_consensus.cc:2749] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:40.719931 2093 ts_tablet_manager.cc:1916] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:40.731328 1883 ts_tablet_manager.cc:1434] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.104s user 0.006s sys 0.000s
I20260501 13:57:40.739955 1899 ts_tablet_manager.cc:1929] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:40.740338 1899 log.cc:1199] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/39cd02738e5f4ecfbfa06d5541614ffd
I20260501 13:57:40.742607 2302 raft_consensus.cc:493] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.743369 1882 tablet_bootstrap.cc:654] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.743032 2302 raft_consensus.cc:515] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.746984 2302 leader_election.cc:290] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:40.747556 1563 leader_election.cc:343] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 9a0d04810f2d465ebe2964797ea30538
I20260501 13:57:40.752703 1883 tablet_bootstrap.cc:492] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:40.756724 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_f9da71780b99473393cc5d38e2d4ca2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:40.758018 1883 tablet_bootstrap.cc:654] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:40.758168 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9da71780b99473393cc5d38e2d4ca2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.765967 1123 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 38c3eb388d474d718554775d65f58d78 (table default.Table_392605e427d14287b4fccdd9b198c6a2 [id=d6ca68cf0d954555b422b90e6502d6cc]). Expected version 1 got 0
I20260501 13:57:40.772550 1881 ts_tablet_manager.cc:1434] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.182s user 0.007s sys 0.000s
I20260501 13:57:40.772693 1123 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_b9fc1d60ce8f47738143b74f0a59edf8" } new_table_name: "default.Table_8876cd859ccb476ea8bec2d65b8d0d60" modify_external_catalogs: true
I20260501 13:57:40.774104 1931 raft_consensus.cc:493] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.774629 1931 raft_consensus.cc:515] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.776294 1931 leader_election.cc:290] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:40.778872 1899 ts_tablet_manager.cc:1950] T 39cd02738e5f4ecfbfa06d5541614ffd P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:40.780741 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 205) took 1048 ms. Trace:
I20260501 13:57:40.781023 1899 rpcz_store.cc:276] 0501 13:57:39.732541 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.732668 (+ 127us) service_pool.cc:224] Handling call
0501 13:57:40.613176 (+880508us) ts_tablet_manager.cc:1151] Deleting tablet 39cd02738e5f4ecfbfa06d5541614ffd
0501 13:57:40.613222 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:40.712308 (+ 99086us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.739842 (+ 27534us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.780682 (+ 40840us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":879977}
I20260501 13:57:40.782256 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 39cd02738e5f4ecfbfa06d5541614ffd (table default.Table_6f9bd93befbf45c0a7415d32ce58d7fb [id=edb6ad6e0b444d3b8930c43ce4082a5c]) successfully deleted
W20260501 13:57:40.789608 1418 leader_election.cc:343] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: eeae4369be954d9d9bad846484afb6ff
I20260501 13:57:40.792717 1881 tablet_bootstrap.cc:492] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:40.793121 1899 tablet_replica.cc:333] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:40.793852 1899 raft_consensus.cc:2243] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:40.795933 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_1d3fc1cda88b46c0814a5c4edaa64340"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:40.796438 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715]). Expected version 1 got 0
I20260501 13:57:40.796844 1899 raft_consensus.cc:2272] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:40.797426 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d3fc1cda88b46c0814a5c4edaa64340 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.801656 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 962f5ac4faef48ca9aa3fd09d35e5274 (DEFAULT_TABLE table=default.Table_5fba8e9c07514a4babeda53e157a580f [id=f918c18f956c4c95811c0f0d1918d77e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.802736 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet 90a57bc58e854e469897a04b2d434d06 with delete_type TABLET_DATA_DELETED (replaced by tablet 3223417aa5594ef7942c6d90bdc8711c) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:40.802841 1899 ts_tablet_manager.cc:1916] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:40.805775 1601 tablet_service.cc:1511] Processing CreateTablet for tablet 962f5ac4faef48ca9aa3fd09d35e5274 (DEFAULT_TABLE table=default.Table_5fba8e9c07514a4babeda53e157a580f [id=f918c18f956c4c95811c0f0d1918d77e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.806823 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 962f5ac4faef48ca9aa3fd09d35e5274. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.811553 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 962f5ac4faef48ca9aa3fd09d35e5274. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.812397 1127 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 0c736959df334aee857dbaae0f7a1e1a
I20260501 13:57:40.814700 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c736959df334aee857dbaae0f7a1e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:40 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:40.816253 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c736959df334aee857dbaae0f7a1e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:40 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:40.817287 1463 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c736959df334aee857dbaae0f7a1e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:40 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:40.818454 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet eeae4369be954d9d9bad846484afb6ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:40.822167 2302 raft_consensus.cc:493] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.822831 2302 raft_consensus.cc:515] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.825071 2302 leader_election.cc:290] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:40.829665 1563 leader_election.cc:343] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 4f2448ee6bf14bbc8ec0e1d59ccb7b90
I20260501 13:57:40.837888 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4121e57a3a954fcd87c520271360371c" } new_table_name: "default.Table_56f3ff0cf6ff4fe4974017f9ed722ca5" modify_external_catalogs: true
I20260501 13:57:40.845470 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 962f5ac4faef48ca9aa3fd09d35e5274 (DEFAULT_TABLE table=default.Table_5fba8e9c07514a4babeda53e157a580f [id=f918c18f956c4c95811c0f0d1918d77e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.845583 1463 tablet_service.cc:1511] Processing CreateTablet for tablet bd633c026195429ca69d8041809bee45 (DEFAULT_TABLE table=default.Table_fe9befde21164496ad12d593c8e08660 [id=5212cd2d6a704fd998cf790d9a5b9539]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.846429 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 962f5ac4faef48ca9aa3fd09d35e5274. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.865379 1899 ts_tablet_manager.cc:1929] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:40.866063 2047 ts_tablet_manager.cc:1929] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:40.876309 1131 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 1bb7fdcad3a54317b43bfd58f08e89d0
I20260501 13:57:40.877553 1131 catalog_manager.cc:6741] ReplaceTablet: tablet 1bb7fdcad3a54317b43bfd58f08e89d0 deleted and replaced by tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9
I20260501 13:57:40.878221 1609 tablet_service.cc:1511] Processing CreateTablet for tablet bd633c026195429ca69d8041809bee45 (DEFAULT_TABLE table=default.Table_fe9befde21164496ad12d593c8e08660 [id=5212cd2d6a704fd998cf790d9a5b9539]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.878355 1457 tablet_service.cc:1558] Processing DeleteTablet for tablet 1bb7fdcad3a54317b43bfd58f08e89d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:40.879256 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd633c026195429ca69d8041809bee45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.880970 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 1bb7fdcad3a54317b43bfd58f08e89d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:40.880537 1124 catalog_manager.cc:5671] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:40.881794 1124 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715]). Expected version 1 got 0
I20260501 13:57:40.882452 1124 catalog_manager.cc:5671] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:40.883533 1124 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (table default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf]). Expected version 1 got 0
I20260501 13:57:40.883936 1130 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 1ab056da506f454984a07026341c534b
I20260501 13:57:40.884472 1124 catalog_manager.cc:5671] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:40.885529 1124 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 9018d70c734a429aa59565ad67a46acc (table default.Table_a7b6759170be4cce860b79126e7bbc29 [id=454925118d6943458e68bb0089610a15]). Expected version 2 got 0
I20260501 13:57:40.886147 1124 catalog_manager.cc:5671] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:40.887282 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ab056da506f454984a07026341c534b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:40 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:40.887548 1457 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ab056da506f454984a07026341c534b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:40 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:40.879786 1755 tablet_service.cc:1558] Processing DeleteTablet for tablet 1bb7fdcad3a54317b43bfd58f08e89d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:40.887527 1124 catalog_manager.cc:5671] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:40.886366 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ab056da506f454984a07026341c534b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:40 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:40.888533 1124 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 6650472ec4a449c48c7957a9bbec88e3 (table default.Table_0d92e7b95c884bd5970e291f4189b0d9 [id=774a7eb90ba449b6aef225d819678439]). Expected version 1 got 0
I20260501 13:57:40.888630 1899 log.cc:1199] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/67b0675398d04be08bd29638b4ba2083
I20260501 13:57:40.889365 1899 ts_tablet_manager.cc:1950] T 67b0675398d04be08bd29638b4ba2083 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:40.891212 2282 raft_consensus.cc:493] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.895213 2282 raft_consensus.cc:515] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.896986 2282 leader_election.cc:290] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:40.899013 1882 tablet_bootstrap.cc:492] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:40.899196 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2ac522f2e4c4ab48a9b7d211c82ed9e" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:40.899446 1882 ts_tablet_manager.cc:1403] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.258s user 0.036s sys 0.028s
I20260501 13:57:40.899264 1752 tablet_service.cc:1511] Processing CreateTablet for tablet bd633c026195429ca69d8041809bee45 (DEFAULT_TABLE table=default.Table_fe9befde21164496ad12d593c8e08660 [id=5212cd2d6a704fd998cf790d9a5b9539]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.900327 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd633c026195429ca69d8041809bee45. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:40.900555 1719 leader_election.cc:343] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.901804 1882 raft_consensus.cc:359] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.902123 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_f9cbd6f5e56b422c897fcbd0b6c7e718"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:40.902402 1882 raft_consensus.cc:385] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:40.903574 1882 raft_consensus.cc:740] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
W20260501 13:57:40.903645 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9cbd6f5e56b422c897fcbd0b6c7e718 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.904218 1882 consensus_queue.cc:260] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.906106 1882 ts_tablet_manager.cc:1434] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260501 13:57:40.906949 1882 tablet_bootstrap.cc:492] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:40.908555 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_628b3fff3b534cdf8d44d255ed5f241f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:40.910066 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_628b3fff3b534cdf8d44d255ed5f241f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:40.911864 1882 tablet_bootstrap.cc:654] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.912043 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 204) took 1179 ms. Trace:
I20260501 13:57:40.912350 1899 rpcz_store.cc:276] 0501 13:57:39.732209 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.733287 (+ 1078us) service_pool.cc:224] Handling call
0501 13:57:40.792971 (+1059684us) ts_tablet_manager.cc:1151] Deleting tablet 67b0675398d04be08bd29638b4ba2083
0501 13:57:40.793012 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:40.839553 (+ 46541us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.865259 (+ 25706us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.911982 (+ 46723us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1059178}
I20260501 13:57:40.913424 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 67b0675398d04be08bd29638b4ba2083 (table default.Table_ddd647c4319b431f9d4e874bb528755d [id=d4ac3b610db54cb8aae381cbc882d18e]) successfully deleted
I20260501 13:57:40.914940 1564 leader_election.cc:304] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:40.915874 2327 raft_consensus.cc:2804] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:40.916333 2327 raft_consensus.cc:697] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:40.917054 2327 consensus_queue.cc:237] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.918504 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeae4369be954d9d9bad846484afb6ff" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:40.919063 1484 raft_consensus.cc:2468] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:40.919775 1899 tablet_replica.cc:333] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:40.920295 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeae4369be954d9d9bad846484afb6ff" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:40.920830 1624 raft_consensus.cc:2468] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:40.922580 1881 tablet_bootstrap.cc:654] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:40.922626 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a0d04810f2d465ebe2964797ea30538" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:40.923377 1899 raft_consensus.cc:2243] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:40.923950 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f2448ee6bf14bbc8ec0e1d59ccb7b90" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:40.927244 1899 raft_consensus.cc:2272] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:40.928697 1564 leader_election.cc:304] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:40.929356 2344 raft_consensus.cc:2804] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:40.929675 2344 raft_consensus.cc:493] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:40.929973 2344 raft_consensus.cc:3060] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:40.930274 1755 tablet_service.cc:1511] Processing CreateTablet for tablet c04f304fc4454d5283e4720f7249528a (DEFAULT_TABLE table=default.Table_059ab1f09d8443f3a261ec50523dbe80 [id=488d3513454b47ba9eebb59bd6a047f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.931614 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04f304fc4454d5283e4720f7249528a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.936249 1753 tablet_service.cc:1511] Processing CreateTablet for tablet c9944c024418466d997d764eb2f3ad50 (DEFAULT_TABLE table=default.Table_f845f5c6881e4f2587c0decd568ffabc [id=47281c712523461c8680efe3d489aa33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.937300 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9944c024418466d997d764eb2f3ad50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.942418 1461 tablet_service.cc:1511] Processing CreateTablet for tablet c9944c024418466d997d764eb2f3ad50 (DEFAULT_TABLE table=default.Table_f845f5c6881e4f2587c0decd568ffabc [id=47281c712523461c8680efe3d489aa33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.943114 2047 log.cc:1199] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/b507a4d4c9c54bc68d92d9286794a264
I20260501 13:57:40.943970 2047 ts_tablet_manager.cc:1950] T b507a4d4c9c54bc68d92d9286794a264 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:40.944105 1457 tablet_service.cc:1511] Processing CreateTablet for tablet c04f304fc4454d5283e4720f7249528a (DEFAULT_TABLE table=default.Table_059ab1f09d8443f3a261ec50523dbe80 [id=488d3513454b47ba9eebb59bd6a047f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.946576 1462 tablet_service.cc:1511] Processing CreateTablet for tablet eec4bce26de14ea1915faea94490d665 (DEFAULT_TABLE table=default.Table_a1116dca3a314f4b8508c62da88fbdc9 [id=5668d18e35c240408ffff27a4e50c843]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.949316 1756 tablet_service.cc:1511] Processing CreateTablet for tablet eec4bce26de14ea1915faea94490d665 (DEFAULT_TABLE table=default.Table_a1116dca3a314f4b8508c62da88fbdc9 [id=5668d18e35c240408ffff27a4e50c843]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.950270 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec4bce26de14ea1915faea94490d665. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:40.955916 1564 leader_election.cc:343] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.956328 1564 leader_election.cc:304] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:40.960527 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd633c026195429ca69d8041809bee45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.961843 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec4bce26de14ea1915faea94490d665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.963168 1899 ts_tablet_manager.cc:1916] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:40.963016 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9944c024418466d997d764eb2f3ad50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.964625 2344 raft_consensus.cc:515] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.966859 1457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04f304fc4454d5283e4720f7249528a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.967690 2328 raft_consensus.cc:2749] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:40.968401 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 641b63b974bf48b9bc40e341f4dee7da (DEFAULT_TABLE table=default.Table_5bb0715c4b1e4e78bbcd523962cf53d7 [id=9829f975be004514bf9f6862fab70fa3]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:40.968652 1564 leader_election.cc:343] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:40.968967 1564 leader_election.cc:304] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:40.969347 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641b63b974bf48b9bc40e341f4dee7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.974787 2274 raft_consensus.cc:2749] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:40.978056 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 641b63b974bf48b9bc40e341f4dee7da (DEFAULT_TABLE table=default.Table_5bb0715c4b1e4e78bbcd523962cf53d7 [id=9829f975be004514bf9f6862fab70fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.979167 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641b63b974bf48b9bc40e341f4dee7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:40.983740 2282 raft_consensus.cc:493] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:40.984203 2282 raft_consensus.cc:515] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:40.984954 1602 tablet_service.cc:1511] Processing CreateTablet for tablet c04f304fc4454d5283e4720f7249528a (DEFAULT_TABLE table=default.Table_059ab1f09d8443f3a261ec50523dbe80 [id=488d3513454b47ba9eebb59bd6a047f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.985900 2282 leader_election.cc:290] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:40.987376 1608 tablet_service.cc:1511] Processing CreateTablet for tablet c9944c024418466d997d764eb2f3ad50 (DEFAULT_TABLE table=default.Table_f845f5c6881e4f2587c0decd568ffabc [id=47281c712523461c8680efe3d489aa33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.988978 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 96875c3b0f7f4421ab5d9f65ba4260a2 (DEFAULT_TABLE table=default.Table_43f068125a9243dbadd21b551da3226b [id=845c47589e074f23b8fee46ab4df5cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.990355 1603 tablet_service.cc:1511] Processing CreateTablet for tablet eec4bce26de14ea1915faea94490d665 (DEFAULT_TABLE table=default.Table_a1116dca3a314f4b8508c62da88fbdc9 [id=5668d18e35c240408ffff27a4e50c843]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:40.991636 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 641b63b974bf48b9bc40e341f4dee7da (DEFAULT_TABLE table=default.Table_5bb0715c4b1e4e78bbcd523962cf53d7 [id=9829f975be004514bf9f6862fab70fa3]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:40.999197 1563 leader_election.cc:343] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: eeae4369be954d9d9bad846484afb6ff
I20260501 13:57:41.001123 1421 leader_election.cc:304] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:41.004663 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96e937b3a0b5451e8eaf4d5c47c17620" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:41.006196 1719 leader_election.cc:343] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:41.006750 2282 raft_consensus.cc:493] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:41.007033 1610 tablet_service.cc:1511] Processing CreateTablet for tablet 96875c3b0f7f4421ab5d9f65ba4260a2 (DEFAULT_TABLE table=default.Table_43f068125a9243dbadd21b551da3226b [id=845c47589e074f23b8fee46ab4df5cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.007164 2282 raft_consensus.cc:515] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:41.007736 1421 consensus_peers.cc:597] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 -> Peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Couldn't send request to peer 8f90b7ff170b4eadb27abc9ef866ffcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ee580bb9d3843f9bbeae8eef2c347ad. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:41.008363 2344 leader_election.cc:290] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:41.008899 2282 leader_election.cc:290] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
W20260501 13:57:41.010207 1119 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 81daa397384d4afb816071ee665a1dcd (table default.Table_ecd622b82fa9463198fe36afb1458343 [id=7ab4b36ba8b8496ea524426efa573d33], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 81daa397384d4afb816071ee665a1dcd consensus state has no leader
I20260501 13:57:41.010790 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96875c3b0f7f4421ab5d9f65ba4260a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.016499 1123 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6ee51a9fc8ae4340b14c506cf9c450c2" } modify_external_catalogs: true
I20260501 13:57:41.017030 1123 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6ee51a9fc8ae4340b14c506cf9c450c2" } modify_external_catalogs: true
I20260501 13:57:41.020191 1899 ts_tablet_manager.cc:1929] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:41.020390 1881 tablet_bootstrap.cc:492] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:41.020541 1899 log.cc:1199] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/9abb39c8b44d4b4c8b69751a4cdd6668
I20260501 13:57:41.020763 1881 ts_tablet_manager.cc:1403] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.228s user 0.009s sys 0.000s
I20260501 13:57:41.021131 1899 ts_tablet_manager.cc:1950] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:41.026654 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51ba7b8a7f4d4ec6ba4af775596d3b59" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:41.027901 1719 leader_election.cc:343] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:41.030437 2369 raft_consensus.cc:2804] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:41.030898 2369 raft_consensus.cc:493] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:41.031255 2369 raft_consensus.cc:3060] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:41.036113 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2ac522f2e4c4ab48a9b7d211c82ed9e" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:41.036417 1753 tablet_service.cc:1511] Processing CreateTablet for tablet 96875c3b0f7f4421ab5d9f65ba4260a2 (DEFAULT_TABLE table=default.Table_43f068125a9243dbadd21b551da3226b [id=845c47589e074f23b8fee46ab4df5cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.037451 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96875c3b0f7f4421ab5d9f65ba4260a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.037617 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96e937b3a0b5451e8eaf4d5c47c17620" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:41.038820 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51ba7b8a7f4d4ec6ba4af775596d3b59" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:41.041118 1717 leader_election.cc:343] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:41.041488 1717 leader_election.cc:304] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:41.042351 2282 raft_consensus.cc:2749] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:41.043231 1717 leader_election.cc:343] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:41.043583 1717 leader_election.cc:304] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:41.044705 2282 raft_consensus.cc:2749] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:41.045533 1717 leader_election.cc:343] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:41.045951 1717 leader_election.cc:304] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:41.046809 2282 raft_consensus.cc:2749] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:41.051563 1881 raft_consensus.cc:359] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.052202 1881 raft_consensus.cc:385] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:41.052500 1881 raft_consensus.cc:740] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:41.054958 2369 raft_consensus.cc:515] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.058060 1761 tablet_service.cc:1511] Processing CreateTablet for tablet c4ce8d198b2b40978a67aa0800dfc47e (DEFAULT_TABLE table=default.Table_2563926cdaef4f91a592f1261851a6b9 [id=3bf8bda1ae3e4015b98a9fb7167d2176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.059116 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ce8d198b2b40978a67aa0800dfc47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.061062 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 202) took 1332 ms. Trace:
I20260501 13:57:41.061398 1899 rpcz_store.cc:276] 0501 13:57:39.728941 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.733870 (+ 4929us) service_pool.cc:224] Handling call
0501 13:57:40.919631 (+1185761us) ts_tablet_manager.cc:1151] Deleting tablet 9abb39c8b44d4b4c8b69751a4cdd6668
0501 13:57:40.919671 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:40.991190 (+ 71519us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.020074 (+ 28884us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.060989 (+ 40915us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1185289}
I20260501 13:57:41.062511 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 9abb39c8b44d4b4c8b69751a4cdd6668 (table default.Table_2a23504aef0e48beaf3541c53fda054e [id=0e61d9e75f4c4b20a849a0540c83bd18]) successfully deleted
I20260501 13:57:41.063975 1462 tablet_service.cc:1511] Processing CreateTablet for tablet a4f7d7867fbd4032869e9448f4993bbc (DEFAULT_TABLE table=default.Table_bc6341b170b3425eb49b5cc95ed40eb2 [id=ea1625f33e304e3cadc9ab5bd843f9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.064777 1899 tablet_replica.cc:333] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
W20260501 13:57:41.064879 1421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 56663.7us
I20260501 13:57:41.067593 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04f304fc4454d5283e4720f7249528a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.068203 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96875c3b0f7f4421ab5d9f65ba4260a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.068744 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9944c024418466d997d764eb2f3ad50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.072816 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641b63b974bf48b9bc40e341f4dee7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.067575 2093 ts_tablet_manager.cc:1929] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:41.075932 2093 log.cc:1199] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/b507a4d4c9c54bc68d92d9286794a264
I20260501 13:57:41.076644 2093 ts_tablet_manager.cc:1950] T b507a4d4c9c54bc68d92d9286794a264 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:41.076923 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec4bce26de14ea1915faea94490d665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.068380 1899 raft_consensus.cc:2243] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:41.083627 1459 tablet_service.cc:1511] Processing CreateTablet for tablet c4ce8d198b2b40978a67aa0800dfc47e (DEFAULT_TABLE table=default.Table_2563926cdaef4f91a592f1261851a6b9 [id=3bf8bda1ae3e4015b98a9fb7167d2176]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:41.084043 1564 consensus_peers.cc:597] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:41.086699 1899 raft_consensus.cc:2272] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:41.088460 1899 ts_tablet_manager.cc:1916] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.094964 1601 tablet_service.cc:1511] Processing CreateTablet for tablet a4f7d7867fbd4032869e9448f4993bbc (DEFAULT_TABLE table=default.Table_bc6341b170b3425eb49b5cc95ed40eb2 [id=ea1625f33e304e3cadc9ab5bd843f9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.096690 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f7d7867fbd4032869e9448f4993bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.104362 1883 tablet_bootstrap.cc:492] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:41.104847 1883 ts_tablet_manager.cc:1403] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.352s user 0.010s sys 0.000s
I20260501 13:57:41.105104 1608 tablet_service.cc:1511] Processing CreateTablet for tablet c4ce8d198b2b40978a67aa0800dfc47e (DEFAULT_TABLE table=default.Table_2563926cdaef4f91a592f1261851a6b9 [id=3bf8bda1ae3e4015b98a9fb7167d2176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.106050 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ce8d198b2b40978a67aa0800dfc47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.107102 2344 consensus_queue.cc:1048] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:41.121186 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6bab24ce4e874b7abcaf569c239b41ea" } new_table_name: "default.Table_ed5738206c234f44b970d6edc256b193" modify_external_catalogs: true
I20260501 13:57:41.123471 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_9f3c5df0faf64dedabec83d1c7d3dd82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:41.124914 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f3c5df0faf64dedabec83d1c7d3dd82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:41.127496 1125 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_430d0393c555431e8d75edfd188a80d6" } modify_external_catalogs: true
I20260501 13:57:41.128015 1125 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_430d0393c555431e8d75edfd188a80d6" } modify_external_catalogs: true
I20260501 13:57:41.103602 1881 consensus_queue.cc:260] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.132759 1753 tablet_service.cc:1511] Processing CreateTablet for tablet a4f7d7867fbd4032869e9448f4993bbc (DEFAULT_TABLE table=default.Table_bc6341b170b3425eb49b5cc95ed40eb2 [id=ea1625f33e304e3cadc9ab5bd843f9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.133526 1881 ts_tablet_manager.cc:1434] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.113s user 0.006s sys 0.001s
I20260501 13:57:41.133994 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f7d7867fbd4032869e9448f4993bbc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:41.137451 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef622e34d46a4b19a59e916d22c2b18b consensus state has no leader
I20260501 13:57:41.141664 1461 tablet_service.cc:1511] Processing CreateTablet for tablet df282981a16d479b81696b8a18f67907 (DEFAULT_TABLE table=default.Table_6ab4c80251da40a783ac60f1b02b941d [id=519d1e782753453695a6c845185faad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.145870 1610 tablet_service.cc:1511] Processing CreateTablet for tablet df282981a16d479b81696b8a18f67907 (DEFAULT_TABLE table=default.Table_6ab4c80251da40a783ac60f1b02b941d [id=519d1e782753453695a6c845185faad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.147015 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df282981a16d479b81696b8a18f67907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.152864 1755 tablet_service.cc:1511] Processing CreateTablet for tablet df282981a16d479b81696b8a18f67907 (DEFAULT_TABLE table=default.Table_6ab4c80251da40a783ac60f1b02b941d [id=519d1e782753453695a6c845185faad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.154027 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeae4369be954d9d9bad846484afb6ff" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:41.154342 1882 tablet_bootstrap.cc:492] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:41.154735 1882 ts_tablet_manager.cc:1403] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.248s user 0.014s sys 0.010s
I20260501 13:57:41.154853 1627 raft_consensus.cc:2393] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 993b9ba18d8547f8af43167e727a6a24 in current term 1: Already voted for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in this term.
I20260501 13:57:41.157564 1882 raft_consensus.cc:359] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:41.157891 1418 leader_election.cc:343] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: eeae4369be954d9d9bad846484afb6ff
I20260501 13:57:41.158190 1882 raft_consensus.cc:385] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:41.160789 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 81230beeff8d434aa2dac467c35d9703 (DEFAULT_TABLE table=default.Table_641d4d711c4d49319daa2cb0cf2714c6 [id=535a372705e4460ab657255e3f1be962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.161017 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 209) took 1480 ms. Trace:
I20260501 13:57:41.161574 1421 leader_election.cc:304] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:41.161401 2093 rpcz_store.cc:276] 0501 13:57:39.680712 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.680830 (+ 118us) service_pool.cc:224] Handling call
0501 13:57:40.643508 (+962678us) ts_tablet_manager.cc:1151] Deleting tablet b507a4d4c9c54bc68d92d9286794a264
0501 13:57:40.643549 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:40.733765 (+ 90216us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.067453 (+333688us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.160934 (+ 93481us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":962140}
I20260501 13:57:41.162439 2093 tablet_replica.cc:333] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:41.162416 2370 raft_consensus.cc:2749] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260501 13:57:41.163012 2369 leader_election.cc:290] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:41.163152 2093 raft_consensus.cc:2243] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:41.170652 1899 ts_tablet_manager.cc:1929] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:41.171034 1899 log.cc:1199] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/e0356b19f0aa4197a41cc66d4440bffb
I20260501 13:57:41.171655 1899 ts_tablet_manager.cc:1950] T e0356b19f0aa4197a41cc66d4440bffb P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:41.173939 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet b507a4d4c9c54bc68d92d9286794a264 (table default.Table_83779be1d30c4839999b9217ce2e4a20 [id=36cedf51b8a8402eb9488172828f7eb8]) successfully deleted
I20260501 13:57:41.182590 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eeae4369be954d9d9bad846484afb6ff" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:41.183315 1484 raft_consensus.cc:2393] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f90b7ff170b4eadb27abc9ef866ffcf in current term 1: Already voted for candidate 993b9ba18d8547f8af43167e727a6a24 in this term.
I20260501 13:57:41.186177 1752 tablet_service.cc:1511] Processing CreateTablet for tablet 81230beeff8d434aa2dac467c35d9703 (DEFAULT_TABLE table=default.Table_641d4d711c4d49319daa2cb0cf2714c6 [id=535a372705e4460ab657255e3f1be962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.187537 1460 tablet_service.cc:1511] Processing CreateTablet for tablet f28db04617a449af8d87a554fb63e1e4 (DEFAULT_TABLE table=default.Table_291632318d004a2d91a9f8d79137ef4d [id=4668751f7bc649019547c38d0f8254c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.190233 1754 tablet_service.cc:1511] Processing CreateTablet for tablet f28db04617a449af8d87a554fb63e1e4 (DEFAULT_TABLE table=default.Table_291632318d004a2d91a9f8d79137ef4d [id=4668751f7bc649019547c38d0f8254c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.191684 2390 tablet.cc:1724] T cf342e482ca04727a10cf79519549b69 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:41.193856 1464 tablet_service.cc:1511] Processing CreateTablet for tablet ef191926320a4e0fa8bd94e47e9b9a86 (DEFAULT_TABLE table=default.Table_66a4e496efa34deaa6c793a210a510ff [id=e8b19350d702499a82f41a6cce14dbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.194923 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81230beeff8d434aa2dac467c35d9703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.202906 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 201) took 1889 ms. Trace:
I20260501 13:57:41.204365 2047 rpcz_store.cc:276] 0501 13:57:39.313898 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.314008 (+ 110us) service_pool.cc:224] Handling call
0501 13:57:40.420769 (+1106761us) ts_tablet_manager.cc:1151] Deleting tablet b507a4d4c9c54bc68d92d9286794a264
0501 13:57:40.420812 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:40.763831 (+343019us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:40.865867 (+102036us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.202825 (+336958us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106183}
I20260501 13:57:41.206302 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet b507a4d4c9c54bc68d92d9286794a264 (table default.Table_83779be1d30c4839999b9217ce2e4a20 [id=36cedf51b8a8402eb9488172828f7eb8]) successfully deleted
I20260501 13:57:41.207249 2047 tablet_replica.cc:333] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:41.207988 2047 raft_consensus.cc:2243] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:41.208819 2047 pending_rounds.cc:70] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: Trying to abort 1 pending ops.
I20260501 13:57:41.209089 2047 pending_rounds.cc:77] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229249438121984 op_type: NO_OP noop_request { }
I20260501 13:57:41.209484 2047 raft_consensus.cc:2889] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:41.209852 2047 raft_consensus.cc:2272] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:41.212225 2047 ts_tablet_manager.cc:1916] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.212821 1757 tablet_service.cc:1511] Processing CreateTablet for tablet ef191926320a4e0fa8bd94e47e9b9a86 (DEFAULT_TABLE table=default.Table_66a4e496efa34deaa6c793a210a510ff [id=e8b19350d702499a82f41a6cce14dbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.213152 1881 tablet_bootstrap.cc:492] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:41.204116 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ce8d198b2b40978a67aa0800dfc47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.214303 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 00e64653da9847cbb1fab02e002993b5 (DEFAULT_TABLE table=default.Table_230433f32f184024be1ec4c52d7acea8 [id=3901e3997c0a4f93bf3925f6dc664d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.215679 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e64653da9847cbb1fab02e002993b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.218369 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 00e64653da9847cbb1fab02e002993b5 (DEFAULT_TABLE table=default.Table_230433f32f184024be1ec4c52d7acea8 [id=3901e3997c0a4f93bf3925f6dc664d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.219709 1458 tablet_service.cc:1511] Processing CreateTablet for tablet 62d7ad7aa5ec4f79a2294023e17766d2 (DEFAULT_TABLE table=default.Table_ef8d1ad25b3a469cb3d3840b998178d8 [id=c3c9a1fa404a4d1684a2edc8b13888f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.220831 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d7ad7aa5ec4f79a2294023e17766d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.223912 1881 tablet_bootstrap.cc:654] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:41.229178 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df282981a16d479b81696b8a18f67907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.238934 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81230beeff8d434aa2dac467c35d9703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.240231 1455 tablet_service.cc:1511] Processing CreateTablet for tablet fb4f00ca29094b9b949c6a5c2718fb02 (DEFAULT_TABLE table=default.Table_59bb2c3a0a8247d98ad73d3a2a21b1d6 [id=64fb279932bb4502817d27bbc9115542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.241305 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 62d7ad7aa5ec4f79a2294023e17766d2 (DEFAULT_TABLE table=default.Table_ef8d1ad25b3a469cb3d3840b998178d8 [id=c3c9a1fa404a4d1684a2edc8b13888f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.242259 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d7ad7aa5ec4f79a2294023e17766d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.242916 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28db04617a449af8d87a554fb63e1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.243497 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef191926320a4e0fa8bd94e47e9b9a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.244694 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 206) took 1416 ms. Trace:
I20260501 13:57:41.245014 1899 rpcz_store.cc:276] 0501 13:57:39.828690 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.828809 (+ 119us) service_pool.cc:224] Handling call
0501 13:57:41.064630 (+1235821us) ts_tablet_manager.cc:1151] Deleting tablet e0356b19f0aa4197a41cc66d4440bffb
0501 13:57:41.064672 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.126250 (+ 61578us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.170529 (+ 44279us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.244638 (+ 74109us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1235300}
I20260501 13:57:41.245965 1899 tablet_replica.cc:333] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:41.246896 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e64653da9847cbb1fab02e002993b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.250727 1899 raft_consensus.cc:2243] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:41.251189 1899 raft_consensus.cc:2272] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:41.251317 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet e0356b19f0aa4197a41cc66d4440bffb (table default.Table_5be69bddaf9d4f6a8e87137a8a46e223 [id=d32e41f59f7d4b45842ea1b263eefbb6]) successfully deleted
I20260501 13:57:41.252784 1899 ts_tablet_manager.cc:1916] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.261543 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4f00ca29094b9b949c6a5c2718fb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.264127 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28db04617a449af8d87a554fb63e1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.266713 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f7d7867fbd4032869e9448f4993bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.267582 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef191926320a4e0fa8bd94e47e9b9a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.278918 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df282981a16d479b81696b8a18f67907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.284778 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_31b7fbe1002b47b0adb3e3abb1517807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:41.286289 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31b7fbe1002b47b0adb3e3abb1517807 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:41.301349 1753 tablet_service.cc:1511] Processing CreateTablet for tablet fb4f00ca29094b9b949c6a5c2718fb02 (DEFAULT_TABLE table=default.Table_59bb2c3a0a8247d98ad73d3a2a21b1d6 [id=64fb279932bb4502817d27bbc9115542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.158425 1882 raft_consensus.cc:740] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:41.302386 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4f00ca29094b9b949c6a5c2718fb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.175609 1602 tablet_service.cc:1511] Processing CreateTablet for tablet 81230beeff8d434aa2dac467c35d9703 (DEFAULT_TABLE table=default.Table_641d4d711c4d49319daa2cb0cf2714c6 [id=535a372705e4460ab657255e3f1be962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.302189 1883 raft_consensus.cc:359] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.302838 1883 raft_consensus.cc:385] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:41.303081 1883 raft_consensus.cc:740] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:41.303220 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81230beeff8d434aa2dac467c35d9703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.302989 1882 consensus_queue.cc:260] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.303725 1883 consensus_queue.cc:260] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.304466 1882 ts_tablet_manager.cc:1434] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.149s user 0.005s sys 0.000s
I20260501 13:57:41.305367 1882 tablet_bootstrap.cc:492] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:41.307320 2328 consensus_queue.cc:1048] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:41.181231 1564 consensus_peers.cc:597] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260501 13:57:41.311646 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 130688us
I20260501 13:57:41.312871 1882 tablet_bootstrap.cc:654] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:41.212533 2344 consensus_queue.cc:1048] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:41.257141 2302 consensus_queue.cc:1048] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:41.315075 1608 tablet_service.cc:1511] Processing CreateTablet for tablet ef191926320a4e0fa8bd94e47e9b9a86 (DEFAULT_TABLE table=default.Table_66a4e496efa34deaa6c793a210a510ff [id=e8b19350d702499a82f41a6cce14dbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.321122 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef191926320a4e0fa8bd94e47e9b9a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.324177 2389 tablet.cc:1724] T cf342e482ca04727a10cf79519549b69 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:41.315447 1564 leader_election.cc:304] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:41.315681 1603 tablet_service.cc:1511] Processing CreateTablet for tablet 62d7ad7aa5ec4f79a2294023e17766d2 (DEFAULT_TABLE table=default.Table_ef8d1ad25b3a469cb3d3840b998178d8 [id=c3c9a1fa404a4d1684a2edc8b13888f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.327479 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d7ad7aa5ec4f79a2294023e17766d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.330511 2093 raft_consensus.cc:2272] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:41.315785 1609 tablet_service.cc:1511] Processing CreateTablet for tablet fb4f00ca29094b9b949c6a5c2718fb02 (DEFAULT_TABLE table=default.Table_59bb2c3a0a8247d98ad73d3a2a21b1d6 [id=64fb279932bb4502817d27bbc9115542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.315066 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 00e64653da9847cbb1fab02e002993b5 (DEFAULT_TABLE table=default.Table_230433f32f184024be1ec4c52d7acea8 [id=3901e3997c0a4f93bf3925f6dc664d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.331497 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e64653da9847cbb1fab02e002993b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.332438 2093 ts_tablet_manager.cc:1916] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.333928 2344 raft_consensus.cc:2749] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260501 13:57:41.316849 1610 tablet_service.cc:1511] Processing CreateTablet for tablet f28db04617a449af8d87a554fb63e1e4 (DEFAULT_TABLE table=default.Table_291632318d004a2d91a9f8d79137ef4d [id=4668751f7bc649019547c38d0f8254c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.339143 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28db04617a449af8d87a554fb63e1e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:41.339635 1564 consensus_peers.cc:597] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:41.340008 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 233e5177f45b4188a64b17dcbe67e770 (DEFAULT_TABLE table=default.Table_e7f9e5733a384b468efb94f393f29328 [id=54d99bda7f334ac89ed3c3292d2a7cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.340291 2344 consensus_queue.cc:1048] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:41.340888 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 233e5177f45b4188a64b17dcbe67e770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.317894 2328 consensus_queue.cc:1048] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:41.350728 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4f00ca29094b9b949c6a5c2718fb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.359201 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 233e5177f45b4188a64b17dcbe67e770 (DEFAULT_TABLE table=default.Table_e7f9e5733a384b468efb94f393f29328 [id=54d99bda7f334ac89ed3c3292d2a7cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.360323 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 233e5177f45b4188a64b17dcbe67e770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.245951 2391 tablet.cc:1724] T cf342e482ca04727a10cf79519549b69 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 13:57:41.393741 1117 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.189:45897, user_credentials={real_user=slave}} blocked reactor thread for 92615.9us
I20260501 13:57:41.399451 1881 tablet_bootstrap.cc:492] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:41.399842 1881 ts_tablet_manager.cc:1403] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.187s user 0.007s sys 0.001s
I20260501 13:57:41.412808 1127 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (table default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf]). Expected version 1 got 0
I20260501 13:57:41.421456 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet 863a10cfd14c4694b7c6141c0851167a with delete_type TABLET_DATA_DELETED (replaced by tablet 6f6cd353ca044465a6692656412e7d82) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:41.426662 1881 raft_consensus.cc:359] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.427325 1881 raft_consensus.cc:385] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:41.427584 1881 raft_consensus.cc:740] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:41.428164 1881 consensus_queue.cc:260] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.430192 1460 tablet_service.cc:1558] Processing DeleteTablet for tablet eeae4369be954d9d9bad846484afb6ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:31 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:41.430547 1881 ts_tablet_manager.cc:1434] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
I20260501 13:57:41.431948 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 233e5177f45b4188a64b17dcbe67e770 (DEFAULT_TABLE table=default.Table_e7f9e5733a384b468efb94f393f29328 [id=54d99bda7f334ac89ed3c3292d2a7cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:41.432642 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:41.435187 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4fe27772a87d4e42840a3ca5f2aed6d6" } new_table_name: "default.Table_bc194cec5138404eb7ba98e3bfb0ea5d" modify_external_catalogs: true
I20260501 13:57:41.438143 1603 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ba63c153ef04db891cae46a71439455 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c1e0c30dce488baed9e046d90f0ba4) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:41.439517 1603 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a0d04810f2d465ebe2964797ea30538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:41.440076 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet 863a10cfd14c4694b7c6141c0851167a with delete_type TABLET_DATA_DELETED (replaced by tablet 6f6cd353ca044465a6692656412e7d82) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:41.447229 1130 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (table default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf]). Expected version 1 got 0
I20260501 13:57:41.448182 1130 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet c33a1f423bdb44f5843daffe8f73e74a (table default.Table_ecd46a159d2448f9ba09fc99e14a22a3 [id=d98d5437c5bc4e44bcfe27983dfa36e6]). Expected version 1 got 0
I20260501 13:57:41.433310 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 233e5177f45b4188a64b17dcbe67e770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:41.448755 2047 ts_tablet_manager.cc:1929] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:41.462445 2047 log.cc:1199] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/2ee580bb9d3843f9bbeae8eef2c347ad
I20260501 13:57:41.463200 2047 ts_tablet_manager.cc:1950] T 2ee580bb9d3843f9bbeae8eef2c347ad P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:41.463958 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 202) took 2149 ms. Trace:
I20260501 13:57:41.464253 2047 rpcz_store.cc:276] 0501 13:57:39.314498 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.314602 (+ 104us) service_pool.cc:224] Handling call
0501 13:57:41.207097 (+1892495us) ts_tablet_manager.cc:1151] Deleting tablet 2ee580bb9d3843f9bbeae8eef2c347ad
0501 13:57:41.207151 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.383594 (+176443us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.448640 (+ 65046us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.463898 (+ 15258us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1891942}
I20260501 13:57:41.465436 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 2ee580bb9d3843f9bbeae8eef2c347ad (table default.Table_abb70b9287914fbc97d3dcf78ac373bf [id=fe9f5542710e48e8bbf9a48e07c412d6]) successfully deleted
I20260501 13:57:41.450977 1881 tablet_bootstrap.cc:492] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:41.468789 1883 ts_tablet_manager.cc:1434] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.364s user 0.004s sys 0.000s
I20260501 13:57:41.469748 1883 tablet_bootstrap.cc:492] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:41.477598 2370 raft_consensus.cc:493] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:41.478039 2370 raft_consensus.cc:515] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.486649 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9c92846d4e4255a1c8ce3835a0a18d" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:41.488212 2370 leader_election.cc:290] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:41.491868 2047 tablet_replica.cc:333] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:41.492627 2047 raft_consensus.cc:2243] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:41.493067 2047 raft_consensus.cc:2272] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:41.495196 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9c92846d4e4255a1c8ce3835a0a18d" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:41.496366 1881 tablet_bootstrap.cc:654] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:41.504266 1564 consensus_peers.cc:597] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:41.506260 1421 leader_election.cc:304] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:41.507477 2047 ts_tablet_manager.cc:1916] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.515102 2370 raft_consensus.cc:2749] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:41.515336 1883 tablet_bootstrap.cc:654] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:41.520675 1899 ts_tablet_manager.cc:1929] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:41.520859 1882 tablet_bootstrap.cc:492] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:41.521039 1899 log.cc:1199] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/d361a2d82ecd456fb43f5e7fe3c83094
I20260501 13:57:41.521319 1882 ts_tablet_manager.cc:1403] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.216s user 0.013s sys 0.003s
I20260501 13:57:41.521617 1899 ts_tablet_manager.cc:1950] T d361a2d82ecd456fb43f5e7fe3c83094 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:41.523738 1882 raft_consensus.cc:359] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.524329 1882 raft_consensus.cc:385] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:41.524567 1882 raft_consensus.cc:740] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:41.541704 1131 master_service.cc:954] ReplaceTablet: received request to replace tablet f52ea8fb8ae0427d804001aed42cd67c from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:41.543365 1123 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 8dbb619863dd405ab20ead6bbebf4d69
I20260501 13:57:41.545981 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet 8dbb619863dd405ab20ead6bbebf4d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:41.546608 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet 8dbb619863dd405ab20ead6bbebf4d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:41.546818 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet 8dbb619863dd405ab20ead6bbebf4d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:41.522324 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 207) took 1687 ms. Trace:
I20260501 13:57:41.554764 1899 rpcz_store.cc:276] 0501 13:57:39.835246 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.835738 (+ 492us) service_pool.cc:224] Handling call
0501 13:57:41.245835 (+1410097us) ts_tablet_manager.cc:1151] Deleting tablet d361a2d82ecd456fb43f5e7fe3c83094
0501 13:57:41.245872 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.291884 (+ 46012us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.520550 (+228666us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.522271 (+ 1721us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1409589}
I20260501 13:57:41.556036 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet d361a2d82ecd456fb43f5e7fe3c83094 (table default.Table_945a3cdaaa824caab261338095683b43 [id=e5c5a3c3a1544b599c53a9b56a6c7297]) successfully deleted
I20260501 13:57:41.556993 1899 tablet_replica.cc:333] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:41.557740 1899 raft_consensus.cc:2243] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:41.558234 1899 raft_consensus.cc:2272] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:41.558322 2047 ts_tablet_manager.cc:1929] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:41.558720 2047 log.cc:1199] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/104708eb7fc0412dad0bd7e14eee9d13
I20260501 13:57:41.559317 2047 ts_tablet_manager.cc:1950] T 104708eb7fc0412dad0bd7e14eee9d13 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:41.560079 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 200) took 2246 ms. Trace:
I20260501 13:57:41.560375 2047 rpcz_store.cc:276] 0501 13:57:39.313579 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.314543 (+ 964us) service_pool.cc:224] Handling call
0501 13:57:41.491698 (+2177155us) ts_tablet_manager.cc:1151] Deleting tablet 104708eb7fc0412dad0bd7e14eee9d13
0501 13:57:41.491757 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.521033 (+ 29276us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.558161 (+ 37128us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.560026 (+ 1865us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12577,"tablet-delete.queue_time_us":2176413}
I20260501 13:57:41.561743 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 104708eb7fc0412dad0bd7e14eee9d13 (table default.Table_4a57dcc1c3194af793a3a0367e766cc6 [id=a921f986b8ac4a948ccb297e1084fccb]) successfully deleted
I20260501 13:57:41.564845 1881 tablet_bootstrap.cc:492] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:41.565256 1881 ts_tablet_manager.cc:1403] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.114s user 0.013s sys 0.004s
I20260501 13:57:41.567591 1881 raft_consensus.cc:359] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.568189 1881 raft_consensus.cc:385] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:41.571278 1881 raft_consensus.cc:740] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:41.574419 2093 ts_tablet_manager.cc:1929] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:41.574808 2093 log.cc:1199] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/920d7f9748ae403d90573d612feeba59
I20260501 13:57:41.575516 2093 ts_tablet_manager.cc:1950] T 920d7f9748ae403d90573d612feeba59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:41.577409 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 210) took 1856 ms. Trace:
I20260501 13:57:41.577702 2093 rpcz_store.cc:276] 0501 13:57:39.721188 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.721726 (+ 538us) service_pool.cc:224] Handling call
0501 13:57:41.162294 (+1440568us) ts_tablet_manager.cc:1151] Deleting tablet 920d7f9748ae403d90573d612feeba59
0501 13:57:41.162335 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.410283 (+247948us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.574296 (+164013us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.577361 (+ 3065us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6848,"tablet-delete.queue_time_us":1440048}
I20260501 13:57:41.577803 1881 consensus_queue.cc:260] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.580506 1881 ts_tablet_manager.cc:1434] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.015s user 0.003s sys 0.004s
I20260501 13:57:41.584545 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 203) took 2250 ms. Trace:
I20260501 13:57:41.584872 2047 rpcz_store.cc:276] 0501 13:57:39.333882 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.338133 (+ 4251us) service_pool.cc:224] Handling call
0501 13:57:41.561366 (+2223233us) ts_tablet_manager.cc:1151] Deleting tablet 0669a229d6d54eacac9889ad18e06f30
0501 13:57:41.561427 (+ 61us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.584490 (+ 23063us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2222539}
W20260501 13:57:41.587199 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 0669a229d6d54eacac9889ad18e06f30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0669a229d6d54eacac9889ad18e06f30 already in progress: creating tablet
I20260501 13:57:41.588246 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 208) took 2050 ms. Trace:
I20260501 13:57:41.588541 2047 rpcz_store.cc:276] 0501 13:57:39.538052 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.573030 (+ 34978us) service_pool.cc:224] Handling call
0501 13:57:41.587889 (+2014859us) ts_tablet_manager.cc:1151] Deleting tablet 68cb7bd1a5aa4419b9c5176a287fe885
0501 13:57:41.587941 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.588185 (+ 244us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2014317}
W20260501 13:57:41.589655 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 68cb7bd1a5aa4419b9c5176a287fe885 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68cb7bd1a5aa4419b9c5176a287fe885 already in progress: creating tablet
I20260501 13:57:41.581807 1881 tablet_bootstrap.cc:492] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:41.594044 1899 ts_tablet_manager.cc:1916] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.591536 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 209) took 2016 ms. Trace:
I20260501 13:57:41.595580 2047 rpcz_store.cc:276] 0501 13:57:39.575286 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.583105 (+ 7819us) service_pool.cc:224] Handling call
0501 13:57:41.591165 (+2008060us) ts_tablet_manager.cc:1151] Deleting tablet 1c630c25ec5c46539936ec545ff4d292
0501 13:57:41.591230 (+ 65us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.591487 (+ 257us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2007551}
W20260501 13:57:41.596813 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 1c630c25ec5c46539936ec545ff4d292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c630c25ec5c46539936ec545ff4d292 already in progress: creating tablet
I20260501 13:57:41.610911 2047 tablet_replica.cc:333] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:41.613554 1881 tablet_bootstrap.cc:654] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:41.614718 2047 raft_consensus.cc:2243] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:41.615247 2047 raft_consensus.cc:2272] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:41.617123 2047 ts_tablet_manager.cc:1916] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.636610 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_8c3198f65f4b4ef78045ab574046fc33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:35.418740 2117 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ca0 after lost signal to thread 2028
W20260501 13:57:41.638020 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c3198f65f4b4ef78045ab574046fc33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:41.638402 2117 kernel_stack_watchdog.cc:198] Thread 2028 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 360ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260501 13:57:34.059193 2126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:57:41.641975 1881 tablet_bootstrap.cc:492] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
W20260501 13:57:41.642076 2133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:57:41.642397 1881 ts_tablet_manager.cc:1403] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.005s
W20260501 13:57:41.643476 2028 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 7.587s user 0.207s sys 0.281s
W20260501 13:57:41.643819 2028 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 7.588s user 0.207s sys 0.281s
I20260501 13:57:41.644728 1881 raft_consensus.cc:359] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.645305 1881 raft_consensus.cc:385] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:41.645535 1881 raft_consensus.cc:740] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
W20260501 13:57:41.646412 2131 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 7593 milliseconds
I20260501 13:57:41.646160 1881 consensus_queue.cc:260] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.651243 1881 ts_tablet_manager.cc:1434] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260501 13:57:41.652405 2327 raft_consensus.cc:493] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:41.652872 2327 raft_consensus.cc:515] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.656858 2047 ts_tablet_manager.cc:1929] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:41.657423 2327 leader_election.cc:290] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:41.659973 2047 log.cc:1199] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/920d7f9748ae403d90573d612feeba59
I20260501 13:57:41.660498 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59508ee6bc6b47959d09f8f5aab89804" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:41.662439 2028 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260501 13:57:41.662743 1564 leader_election.cc:343] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:41.664052 2028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:57:41.660882 2047 ts_tablet_manager.cc:1950] T 920d7f9748ae403d90573d612feeba59 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:41.674583 1125 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet daca20eeafb141f2bf8b4233e0c8ea3b
I20260501 13:57:41.676095 1125 master_service.cc:954] ReplaceTablet: received request to replace tablet 15a1f80cf7d24974b00c6cca7ae5783e from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:41.677726 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59508ee6bc6b47959d09f8f5aab89804" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:41.678287 1780 raft_consensus.cc:2468] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:41.683270 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet daca20eeafb141f2bf8b4233e0c8ea3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:41.683346 1563 leader_election.cc:304] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:41.683910 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet daca20eeafb141f2bf8b4233e0c8ea3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:41.684343 2327 raft_consensus.cc:2804] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:41.684679 2327 raft_consensus.cc:493] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:41.684988 2327 raft_consensus.cc:3060] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:41.685213 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet daca20eeafb141f2bf8b4233e0c8ea3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:41.686813 1882 consensus_queue.cc:260] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.688741 1882 ts_tablet_manager.cc:1434] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.167s user 0.004s sys 0.000s
I20260501 13:57:41.689677 1882 tablet_bootstrap.cc:492] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:41.692173 2327 raft_consensus.cc:515] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.694645 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59508ee6bc6b47959d09f8f5aab89804" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
W20260501 13:57:41.695801 1564 leader_election.cc:343] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:41.707103 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59508ee6bc6b47959d09f8f5aab89804" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:41.707746 1780 raft_consensus.cc:3060] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:41.711346 2327 leader_election.cc:290] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:41.712366 1883 tablet_bootstrap.cc:492] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:41.712752 1883 ts_tablet_manager.cc:1403] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.243s user 0.011s sys 0.000s
I20260501 13:57:41.728021 1780 raft_consensus.cc:2468] T 59508ee6bc6b47959d09f8f5aab89804 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
W20260501 13:57:41.733240 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77c59fc1fdd645af8c3017b6836c8bf4 consensus state has no leader
I20260501 13:57:41.738828 2412 raft_consensus.cc:493] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:41.739269 2412 raft_consensus.cc:515] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.724701 1883 raft_consensus.cc:359] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.741361 2412 leader_election.cc:290] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:41.742034 2028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:57:41.742058 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eb6d30bed2a410db8b62fdff54a5fac" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:41.743153 1564 leader_election.cc:343] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:41.743716 1883 raft_consensus.cc:385] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:41.744033 1883 raft_consensus.cc:740] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:41.746284 1883 consensus_queue.cc:260] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.752621 2028 hybrid_clock.cc:648] HybridClock initialized: now 1777643861752558 us; error 69 us; skew 500 ppm
I20260501 13:57:41.753296 1883 ts_tablet_manager.cc:1434] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20260501 13:57:41.753736 2028 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:57:41.756227 1881 tablet_bootstrap.cc:492] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:41.761406 1883 tablet_bootstrap.cc:492] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:41.767262 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_e2f41fc3135c458e9fda2d60189b4a23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:41.768723 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2f41fc3135c458e9fda2d60189b4a23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:41.773334 1899 ts_tablet_manager.cc:1929] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:41.773728 1899 log.cc:1199] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/e1d7a80ad4c4455a9ddee7a955e32635
I20260501 13:57:41.778088 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_f8e9d88347b54e2da903776319f1c181" } modify_external_catalogs: true
I20260501 13:57:41.778609 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_f8e9d88347b54e2da903776319f1c181" } modify_external_catalogs: true
W20260501 13:57:41.786108 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 49222.3us
I20260501 13:57:41.774389 1899 ts_tablet_manager.cc:1950] T e1d7a80ad4c4455a9ddee7a955e32635 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:41.787773 1563 leader_election.cc:304] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:41.788678 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 208) took 1936 ms. Trace:
I20260501 13:57:41.789095 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eb6d30bed2a410db8b62fdff54a5fac" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:41.789709 1782 raft_consensus.cc:2468] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:41.791299 2409 raft_consensus.cc:2804] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:41.791741 2409 raft_consensus.cc:697] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:41.792364 2396 raft_consensus.cc:493] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:41.792883 2396 raft_consensus.cc:515] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.792510 2409 consensus_queue.cc:237] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.795156 2396 leader_election.cc:290] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:41.795554 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d399de014d74cab8df14f44787ca2f7" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:41.796679 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d399de014d74cab8df14f44787ca2f7" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:41.796901 1719 leader_election.cc:343] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:41.799337 1883 tablet_bootstrap.cc:654] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:41.799435 1717 leader_election.cc:343] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:41.799804 1717 leader_election.cc:304] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:41.802001 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 920d7f9748ae403d90573d612feeba59 (table default.Table_f0801819716e426f92844203c4a06bdc [id=068f5ac17d494d0b8639cbdc07a04597]) successfully deleted
I20260501 13:57:41.802189 2093 tablet_replica.cc:333] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:41.802874 2093 raft_consensus.cc:2243] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:41.803290 2093 raft_consensus.cc:2272] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:41.804876 2093 ts_tablet_manager.cc:1916] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.811908 1882 tablet_bootstrap.cc:654] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:41.788991 1899 rpcz_store.cc:276] 0501 13:57:39.851939 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.852049 (+ 110us) service_pool.cc:224] Handling call
0501 13:57:41.556833 (+1704784us) ts_tablet_manager.cc:1151] Deleting tablet e1d7a80ad4c4455a9ddee7a955e32635
0501 13:57:41.556881 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.708853 (+151972us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.773199 (+ 64346us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.788622 (+ 15423us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1704173}
I20260501 13:57:41.813897 1899 tablet_replica.cc:333] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:41.813911 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet e1d7a80ad4c4455a9ddee7a955e32635 (table default.Table_cf80c30d6b014d56aab7e294515801d5 [id=16dcfdbf585e45b29a6cdc6442d0990b]) successfully deleted
I20260501 13:57:41.814682 1899 raft_consensus.cc:2243] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:41.815167 1899 raft_consensus.cc:2272] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:41.818110 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee76efc1e2204502b61bc5c7892ed856 has no consensus state
I20260501 13:57:41.820003 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 210) took 2098 ms. Trace:
I20260501 13:57:41.820318 2047 rpcz_store.cc:276] 0501 13:57:39.721571 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.734435 (+ 12864us) service_pool.cc:224] Handling call
0501 13:57:41.610737 (+1876302us) ts_tablet_manager.cc:1151] Deleting tablet 920d7f9748ae403d90573d612feeba59
0501 13:57:41.610791 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.631296 (+ 20505us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.656727 (+ 25431us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.819947 (+163220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2995,"tablet-delete.queue_time_us":1869909}
I20260501 13:57:41.821627 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 211) took 1873 ms. Trace:
I20260501 13:57:41.823171 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 920d7f9748ae403d90573d612feeba59 (table default.Table_f0801819716e426f92844203c4a06bdc [id=068f5ac17d494d0b8639cbdc07a04597]) successfully deleted
I20260501 13:57:41.821928 2047 rpcz_store.cc:276] 0501 13:57:39.947978 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.948099 (+ 121us) service_pool.cc:224] Handling call
0501 13:57:41.821229 (+1873130us) ts_tablet_manager.cc:1151] Deleting tablet 51ba7b8a7f4d4ec6ba4af775596d3b59
0501 13:57:41.821290 (+ 61us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.821579 (+ 289us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1872598}
I20260501 13:57:41.824229 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 212) took 1851 ms. Trace:
W20260501 13:57:41.824443 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 already in progress: creating tablet
I20260501 13:57:41.824505 2047 rpcz_store.cc:276] 0501 13:57:39.972617 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.972742 (+ 125us) service_pool.cc:224] Handling call
0501 13:57:41.823881 (+1851139us) ts_tablet_manager.cc:1151] Deleting tablet b125d1a190f947a5b2ef942f04408c51
0501 13:57:41.823933 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.824184 (+ 251us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1850609}
W20260501 13:57:41.825502 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet b125d1a190f947a5b2ef942f04408c51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b125d1a190f947a5b2ef942f04408c51 already in progress: creating tablet
I20260501 13:57:41.826273 2047 tablet_replica.cc:333] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:41.827854 2047 raft_consensus.cc:2243] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:41.828163 1881 tablet_bootstrap.cc:654] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:41.828338 2047 raft_consensus.cc:2272] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:41.830168 2047 ts_tablet_manager.cc:1916] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.830221 1883 tablet_bootstrap.cc:492] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:41.832845 1899 ts_tablet_manager.cc:1916] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.835963 1883 ts_tablet_manager.cc:1403] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.002s
I20260501 13:57:41.838407 1883 raft_consensus.cc:359] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.839112 1883 raft_consensus.cc:385] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:41.839422 1883 raft_consensus.cc:740] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:41.840153 1883 consensus_queue.cc:260] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.841081 2282 raft_consensus.cc:2749] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:41.843997 1883 ts_tablet_manager.cc:1434] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20260501 13:57:41.844925 1883 tablet_bootstrap.cc:492] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:41.850015 1883 tablet_bootstrap.cc:654] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:41.858908 1563 leader_election.cc:304] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:41.860494 2344 raft_consensus.cc:2804] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:41.860893 2344 raft_consensus.cc:493] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:41.861195 2344 raft_consensus.cc:3060] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:41.864602 2391 tablet.cc:1724] T 9018d70c734a429aa59565ad67a46acc P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:41.868610 2390 tablet.cc:1724] T 9018d70c734a429aa59565ad67a46acc P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:41.881413 2344 raft_consensus.cc:515] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.884630 2093 ts_tablet_manager.cc:1929] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:41.884384 1883 tablet_bootstrap.cc:492] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:41.884994 2093 log.cc:1199] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/b98187c7843e42b09d55c851117c4ac8
I20260501 13:57:41.885587 2093 ts_tablet_manager.cc:1950] T b98187c7843e42b09d55c851117c4ac8 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:41.885587 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eb6d30bed2a410db8b62fdff54a5fac" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:41.885977 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_392605e427d14287b4fccdd9b198c6a2" } modify_external_catalogs: true
I20260501 13:57:41.886507 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_392605e427d14287b4fccdd9b198c6a2" } modify_external_catalogs: true
W20260501 13:57:41.887032 1564 leader_election.cc:343] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:41.887880 1883 ts_tablet_manager.cc:1403] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.012s
I20260501 13:57:41.887796 2344 leader_election.cc:290] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:41.890486 1883 raft_consensus.cc:359] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.891214 1883 raft_consensus.cc:385] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:41.891498 1883 raft_consensus.cc:740] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:41.892297 1883 consensus_queue.cc:260] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.895125 1883 ts_tablet_manager.cc:1434] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260501 13:57:41.895304 1778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eb6d30bed2a410db8b62fdff54a5fac" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:41.895928 1778 raft_consensus.cc:3060] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:41.896234 1883 tablet_bootstrap.cc:492] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:41.899348 2047 ts_tablet_manager.cc:1929] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:41.899765 2047 log.cc:1199] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/9c789ca9e3c748d5bc79f4e1b0ea8fd1
I20260501 13:57:41.900468 2047 ts_tablet_manager.cc:1950] T 9c789ca9e3c748d5bc79f4e1b0ea8fd1 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:41.901371 2370 raft_consensus.cc:493] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:41.901800 1883 tablet_bootstrap.cc:654] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:41.901824 2370 raft_consensus.cc:515] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.902901 1778 raft_consensus.cc:2468] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:41.903560 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 211) took 2031 ms. Trace:
I20260501 13:57:41.903877 2370 leader_election.cc:290] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:41.903848 2093 rpcz_store.cc:276] 0501 13:57:39.872103 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.872215 (+ 112us) service_pool.cc:224] Handling call
0501 13:57:41.802042 (+1929827us) ts_tablet_manager.cc:1151] Deleting tablet b98187c7843e42b09d55c851117c4ac8
0501 13:57:41.802080 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.858344 (+ 56264us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.884513 (+ 26169us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.903495 (+ 18982us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1929290}
I20260501 13:57:41.905279 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet b98187c7843e42b09d55c851117c4ac8 (table default.Table_8999f52c517a462a9845800aba73657e [id=e1549f4369a14aceb962e2be6f1a2ea4]) successfully deleted
I20260501 13:57:41.906162 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a0d04810f2d465ebe2964797ea30538" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:41.906843 1629 raft_consensus.cc:2468] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:41.908613 1421 leader_election.cc:304] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:41.909230 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 212) took 1765 ms. Trace:
I20260501 13:57:41.909529 2093 rpcz_store.cc:276] 0501 13:57:40.144094 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.144190 (+ 96us) service_pool.cc:224] Handling call
0501 13:57:41.908891 (+1764701us) ts_tablet_manager.cc:1151] Deleting tablet 51ba7b8a7f4d4ec6ba4af775596d3b59
0501 13:57:41.908933 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.909180 (+ 247us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1764206}
I20260501 13:57:41.910259 2370 raft_consensus.cc:2804] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:41.910691 2370 raft_consensus.cc:493] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:41.910989 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 213) took 1766 ms. Trace:
W20260501 13:57:41.911284 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 already in progress: creating tablet
I20260501 13:57:41.911963 2370 raft_consensus.cc:3060] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:41.912529 1484 raft_consensus.cc:3060] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:41.910401 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 213) took 1906 ms. Trace:
I20260501 13:57:41.915313 2047 rpcz_store.cc:276] 0501 13:57:39.996091 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.996234 (+ 143us) service_pool.cc:224] Handling call
0501 13:57:41.826132 (+1829898us) ts_tablet_manager.cc:1151] Deleting tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1
0501 13:57:41.826182 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.851696 (+ 25514us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.899202 (+ 47506us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.902406 (+ 3204us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1829266}
I20260501 13:57:41.915947 1883 tablet_bootstrap.cc:492] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:41.916489 1883 ts_tablet_manager.cc:1403] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20260501 13:57:41.916680 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 9c789ca9e3c748d5bc79f4e1b0ea8fd1 (table default.Table_2a1ca2e987e243ad9f1c064a28866dd3 [id=ae0dad3b628b454d899e4912c8013d14]) successfully deleted
I20260501 13:57:41.911275 2093 rpcz_store.cc:276] 0501 13:57:40.144373 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.144471 (+ 98us) service_pool.cc:224] Handling call
0501 13:57:41.910694 (+1766223us) ts_tablet_manager.cc:1151] Deleting tablet b125d1a190f947a5b2ef942f04408c51
0501 13:57:41.910732 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.910942 (+ 210us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1765742}
W20260501 13:57:41.918126 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet b125d1a190f947a5b2ef942f04408c51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b125d1a190f947a5b2ef942f04408c51 already in progress: creating tablet
I20260501 13:57:41.918434 2047 tablet_replica.cc:333] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:41.919229 2093 tablet_replica.cc:333] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:41.919584 1883 raft_consensus.cc:359] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.919937 2093 raft_consensus.cc:2243] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:41.920197 1883 raft_consensus.cc:385] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:41.919927 2370 raft_consensus.cc:515] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.920521 2093 raft_consensus.cc:2272] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:41.921988 1883 raft_consensus.cc:740] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:41.922422 2093 ts_tablet_manager.cc:1916] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.923187 1883 consensus_queue.cc:260] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.924968 1883 ts_tablet_manager.cc:1434] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
W20260501 13:57:41.923521 1418 leader_election.cc:343] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 9a0d04810f2d465ebe2964797ea30538
I20260501 13:57:41.925927 2047 raft_consensus.cc:2243] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:41.927954 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a0d04810f2d465ebe2964797ea30538" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:41.928462 1629 raft_consensus.cc:3060] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:41.926867 2047 raft_consensus.cc:2272] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:41.931299 2047 ts_tablet_manager.cc:1916] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:41.938653 1629 raft_consensus.cc:2468] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:41.939385 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_bd1dfe371f64485fb2a9b88dca4dd231" } new_table_name: "default.Table_b29786a52abd4fad9a2337d7c76fbbc4" modify_external_catalogs: true
I20260501 13:57:41.939752 2396 raft_consensus.cc:493] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:41.939847 1421 leader_election.cc:304] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:41.940193 2396 raft_consensus.cc:515] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.943590 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8efee17e2c34caf98c0ede01d2a943a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:41.946018 2028 webserver.cc:492] Webserver started at http://127.1.16.188:41671/ using document root <none> and password file <none>
I20260501 13:57:41.947461 2028 fs_manager.cc:362] Metadata directory not provided
I20260501 13:57:41.947783 2028 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:57:41.948786 1883 tablet_bootstrap.cc:492] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:41.948734 2282 raft_consensus.cc:493] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:41.949246 2282 raft_consensus.cc:515] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.951038 2282 leader_election.cc:290] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:41.961660 1131 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet f52ea8fb8ae0427d804001aed42cd67c
I20260501 13:57:41.967300 1131 catalog_manager.cc:6741] ReplaceTablet: tablet f52ea8fb8ae0427d804001aed42cd67c deleted and replaced by tablet 4802dafc3e6b4874898b7dee694fc76e
I20260501 13:57:41.968066 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet f52ea8fb8ae0427d804001aed42cd67c with delete_type TABLET_DATA_DELETED (replaced by tablet 4802dafc3e6b4874898b7dee694fc76e) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:41.968879 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet f52ea8fb8ae0427d804001aed42cd67c with delete_type TABLET_DATA_DELETED (replaced by tablet 4802dafc3e6b4874898b7dee694fc76e) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:41.970772 1883 tablet_bootstrap.cc:654] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:41.984905 2425 raft_consensus.cc:2804] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:41.985471 2425 raft_consensus.cc:697] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:41.986501 2425 consensus_queue.cc:237] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:41.993386 1899 ts_tablet_manager.cc:1929] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:41.993844 1899 log.cc:1199] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/94abe840b1994c0cb1f7ac7f4ac09cd9
I20260501 13:57:41.996083 2093 ts_tablet_manager.cc:1929] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260501 13:57:41.992815 1418 leader_election.cc:343] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 9a0d04810f2d465ebe2964797ea30538
I20260501 13:57:41.996472 2093 log.cc:1199] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/b8d0f6d1f38b428291dba22015ac6959
I20260501 13:57:41.994215 2370 leader_election.cc:290] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:41.997159 2093 ts_tablet_manager.cc:1950] T b8d0f6d1f38b428291dba22015ac6959 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:41.995380 2047 ts_tablet_manager.cc:1929] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:57:41.998239 2047 log.cc:1199] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/b8d0f6d1f38b428291dba22015ac6959
I20260501 13:57:41.999001 2047 ts_tablet_manager.cc:1950] T b8d0f6d1f38b428291dba22015ac6959 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:42.000931 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 214) took 1485 ms. Trace:
I20260501 13:57:42.001742 1899 ts_tablet_manager.cc:1950] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:42.002301 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8efee17e2c34caf98c0ede01d2a943a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:42.003377 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet f52ea8fb8ae0427d804001aed42cd67c with delete_type TABLET_DATA_DELETED (replaced by tablet 4802dafc3e6b4874898b7dee694fc76e) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:42.005317 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 214) took 1476 ms. Trace:
I20260501 13:57:42.007108 1563 leader_election.cc:304] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.007460 1882 tablet_bootstrap.cc:492] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:42.007982 1882 ts_tablet_manager.cc:1403] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.318s user 0.010s sys 0.010s
I20260501 13:57:42.008306 2302 raft_consensus.cc:2804] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:42.008883 2302 raft_consensus.cc:697] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:42.009717 2302 consensus_queue.cc:237] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:42.004782 1719 leader_election.cc:343] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.011715 1882 raft_consensus.cc:359] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.012529 1882 raft_consensus.cc:385] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:42.012782 1882 raft_consensus.cc:740] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:42.013412 1882 consensus_queue.cc:260] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.013918 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 209) took 2159 ms. Trace:
I20260501 13:57:42.017206 1882 ts_tablet_manager.cc:1434] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20260501 13:57:42.017772 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9169851f1358460d99a48cb6067ab3e3" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:42.019001 1719 leader_election.cc:343] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:42.019574 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 107546us
I20260501 13:57:42.020475 1882 tablet_bootstrap.cc:492] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
W20260501 13:57:42.022955 1118 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet ee76efc1e2204502b61bc5c7892ed856 (table default.Table_bc194cec5138404eb7ba98e3bfb0ea5d [id=ebc44f52a33846f3b66b4c9d498f9651], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ee76efc1e2204502b61bc5c7892ed856 has no consensus state
I20260501 13:57:42.018841 2396 leader_election.cc:290] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:42.014245 1899 rpcz_store.cc:276] 0501 13:57:39.854171 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.854277 (+ 106us) service_pool.cc:224] Handling call
0501 13:57:41.813752 (+1959475us) ts_tablet_manager.cc:1151] Deleting tablet 94abe840b1994c0cb1f7ac7f4ac09cd9
0501 13:57:41.813793 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.875851 (+ 62058us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.993236 (+117385us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.013846 (+ 20610us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1958879}
I20260501 13:57:42.029629 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 (table default.Table_291632318d004a2d91a9f8d79137ef4d [id=4668751f7bc649019547c38d0f8254c4]) successfully deleted
I20260501 13:57:42.001248 2047 rpcz_store.cc:276] 0501 13:57:40.515227 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.515357 (+ 130us) service_pool.cc:224] Handling call
0501 13:57:41.918264 (+1402907us) ts_tablet_manager.cc:1151] Deleting tablet b8d0f6d1f38b428291dba22015ac6959
0501 13:57:41.918324 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.956517 (+ 38193us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.995256 (+ 38739us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.000866 (+ 5610us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1402371}
I20260501 13:57:42.005630 2093 rpcz_store.cc:276] 0501 13:57:40.528789 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.528900 (+ 111us) service_pool.cc:224] Handling call
0501 13:57:41.919064 (+1390164us) ts_tablet_manager.cc:1151] Deleting tablet b8d0f6d1f38b428291dba22015ac6959
0501 13:57:41.919104 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:41.970848 (+ 51744us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:41.995953 (+ 25105us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.005262 (+ 9309us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6130,"tablet-delete.queue_time_us":1382751}
W20260501 13:57:42.030926 1421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 91177.8us
I20260501 13:57:42.031998 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 216) took 1215 ms. Trace:
I20260501 13:57:42.032330 2047 rpcz_store.cc:276] 0501 13:57:40.816916 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.817072 (+ 156us) service_pool.cc:224] Handling call
0501 13:57:42.031623 (+1214551us) ts_tablet_manager.cc:1151] Deleting tablet 0c736959df334aee857dbaae0f7a1e1a
0501 13:57:42.031665 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.031939 (+ 274us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213846}
I20260501 13:57:42.034612 1478 raft_consensus.cc:3060] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:42.037410 2344 consensus_queue.cc:1048] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:42.038417 1899 tablet_replica.cc:333] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:42.030588 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet b8d0f6d1f38b428291dba22015ac6959 (table default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba]) successfully deleted
W20260501 13:57:42.039335 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 0c736959df334aee857dbaae0f7a1e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c736959df334aee857dbaae0f7a1e1a already in progress: creating tablet
I20260501 13:57:42.044602 1899 raft_consensus.cc:2243] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:42.046279 2093 tablet_replica.cc:333] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:42.046813 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet b8d0f6d1f38b428291dba22015ac6959 (table default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba]) successfully deleted
I20260501 13:57:42.047051 2093 raft_consensus.cc:2243] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:42.047474 2093 raft_consensus.cc:2272] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:42.051115 2093 ts_tablet_manager.cc:1916] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:42.051827 1899 raft_consensus.cc:2272] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:42.053630 1899 ts_tablet_manager.cc:1916] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:42.039500 1717 leader_election.cc:343] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.055749 1882 tablet_bootstrap.cc:654] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:42.056038 1717 leader_election.cc:304] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.056933 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_79fb3bc412df49ea8ed51d3a8a6f91f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:42.057439 2424 raft_consensus.cc:2749] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:42.058424 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79fb3bc412df49ea8ed51d3a8a6f91f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:42.062439 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_60c762a708a54d729a3eb53a841dfe12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:42.063098 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_291632318d004a2d91a9f8d79137ef4d" } new_table_name: "default.Table_8a2d4f67afd1427b8c32c5bb223a7148" modify_external_catalogs: true
I20260501 13:57:42.065349 2093 ts_tablet_manager.cc:1929] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:42.065699 2093 log.cc:1199] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/90a57bc58e854e469897a04b2d434d06
I20260501 13:57:42.066290 2093 ts_tablet_manager.cc:1950] T 90a57bc58e854e469897a04b2d434d06 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:42.066864 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 218) took 1189 ms. Trace:
I20260501 13:57:42.066993 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9169851f1358460d99a48cb6067ab3e3" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:42.067198 2047 rpcz_store.cc:276] 0501 13:57:40.877912 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.878096 (+ 184us) service_pool.cc:224] Handling call
0501 13:57:42.058548 (+1180452us) ts_tablet_manager.cc:1151] Deleting tablet 1bb7fdcad3a54317b43bfd58f08e89d0
0501 13:57:42.058593 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.066808 (+ 8215us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1179651}
I20260501 13:57:42.068132 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 215) took 1265 ms. Trace:
I20260501 13:57:42.068828 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 219) took 1181 ms. Trace:
I20260501 13:57:42.068420 2093 rpcz_store.cc:276] 0501 13:57:40.802400 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.802570 (+ 170us) service_pool.cc:224] Handling call
0501 13:57:42.046103 (+1243533us) ts_tablet_manager.cc:1151] Deleting tablet 90a57bc58e854e469897a04b2d434d06
0501 13:57:42.046151 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.059007 (+ 12856us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.065231 (+ 6224us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.068076 (+ 2845us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1242917}
W20260501 13:57:42.069687 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 1bb7fdcad3a54317b43bfd58f08e89d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bb7fdcad3a54317b43bfd58f08e89d0 already in progress: creating tablet
I20260501 13:57:42.069123 2047 rpcz_store.cc:276] 0501 13:57:40.887162 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.887394 (+ 232us) service_pool.cc:224] Handling call
0501 13:57:42.068492 (+1181098us) ts_tablet_manager.cc:1151] Deleting tablet 1ab056da506f454984a07026341c534b
0501 13:57:42.068535 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.068778 (+ 243us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1180544}
W20260501 13:57:42.071841 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 1ab056da506f454984a07026341c534b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ab056da506f454984a07026341c534b already in progress: creating tablet
I20260501 13:57:42.072053 2047 tablet_replica.cc:333] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:42.071343 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 217) took 1255 ms. Trace:
I20260501 13:57:42.072579 2093 rpcz_store.cc:276] 0501 13:57:40.815945 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.816090 (+ 145us) service_pool.cc:224] Handling call
0501 13:57:42.070722 (+1254632us) ts_tablet_manager.cc:1151] Deleting tablet 0c736959df334aee857dbaae0f7a1e1a
0501 13:57:42.070817 (+ 95us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.071233 (+ 416us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254045}
I20260501 13:57:42.074657 2093 tablet_replica.cc:333] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:42.075949 2093 raft_consensus.cc:2243] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:42.076943 2093 raft_consensus.cc:2272] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:42.078852 2028 fs_manager.cc:714] Time spent opening directory manager: real 0.076s user 0.005s sys 0.001s
W20260501 13:57:42.082288 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60c762a708a54d729a3eb53a841dfe12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:42.069648 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 90a57bc58e854e469897a04b2d434d06 (table default.Table_f8e9d88347b54e2da903776319f1c181 [id=b3e0852bf1bc4eedad8b69f7e4cf0785]) successfully deleted
W20260501 13:57:42.084496 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 0c736959df334aee857dbaae0f7a1e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c736959df334aee857dbaae0f7a1e1a already in progress: creating tablet
I20260501 13:57:42.087512 2093 ts_tablet_manager.cc:1916] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:42.090632 2047 raft_consensus.cc:2243] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:42.093330 2047 pending_rounds.cc:70] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: Trying to abort 1 pending ops.
I20260501 13:57:42.093576 2047 pending_rounds.cc:77] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229254451822592 op_type: NO_OP noop_request { }
I20260501 13:57:42.093925 2047 raft_consensus.cc:2889] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:42.094276 2047 raft_consensus.cc:2272] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:42.096453 1882 tablet_bootstrap.cc:492] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
W20260501 13:57:42.096633 1564 consensus_peers.cc:597] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:42.096951 1882 ts_tablet_manager.cc:1403] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.003s
I20260501 13:57:42.099465 1882 raft_consensus.cc:359] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.100293 1882 raft_consensus.cc:385] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:42.100557 1882 raft_consensus.cc:740] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:42.101114 1882 consensus_queue.cc:260] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:42.104152 1717 leader_election.cc:343] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.104435 2047 ts_tablet_manager.cc:1916] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:42.106225 1881 tablet_bootstrap.cc:492] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:42.112716 1717 leader_election.cc:304] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.114522 1881 ts_tablet_manager.cc:1403] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.358s user 0.009s sys 0.001s
I20260501 13:57:42.115556 2437 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:57:42.121225 2424 raft_consensus.cc:2749] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:42.116916 1882 ts_tablet_manager.cc:1434] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260501 13:57:42.123426 2028 fs_manager.cc:730] Time spent opening block manager: real 0.042s user 0.005s sys 0.001s
W20260501 13:57:42.126379 1128 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96e937b3a0b5451e8eaf4d5c47c17620 consensus state has no leader
I20260501 13:57:42.123867 2028 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal
uuid: "50c0932256e74399a537670bb424a1b7"
format_stamp: "Formatted at 2026-05-01 13:57:18 on dist-test-slave-nqxd"
I20260501 13:57:42.126732 2028 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260501 13:57:42.128173 1563 consensus_peers.cc:597] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Couldn't send request to peer 0ed1f9a545474e76a661c32bf11a7fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 863a10cfd14c4694b7c6141c0851167a. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:42.131786 1882 tablet_bootstrap.cc:492] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
W20260501 13:57:42.137533 1119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260501 13:57:42.134488 2093 ts_tablet_manager.cc:1929] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:42.142654 2093 log.cc:1199] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/eeae4369be954d9d9bad846484afb6ff
I20260501 13:57:42.143345 2093 ts_tablet_manager.cc:1950] T eeae4369be954d9d9bad846484afb6ff P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:42.149830 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 218) took 1331 ms. Trace:
W20260501 13:57:42.155442 1119 consensus_peers.cc:597] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b -> Peer 50c0932256e74399a537670bb424a1b7 (127.1.16.188:38815): Couldn't send request to peer 50c0932256e74399a537670bb424a1b7. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38815: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260501 13:57:42.158236 1899 ts_tablet_manager.cc:1929] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:42.158660 1899 log.cc:1199] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/51ba7b8a7f4d4ec6ba4af775596d3b59
I20260501 13:57:42.159279 1899 ts_tablet_manager.cc:1950] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
W20260501 13:57:42.159353 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f28db04617a449af8d87a554fb63e1e4 consensus state has no leader
I20260501 13:57:42.150174 2093 rpcz_store.cc:276] 0501 13:57:40.818216 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.818316 (+ 100us) service_pool.cc:224] Handling call
0501 13:57:42.074341 (+1256025us) ts_tablet_manager.cc:1151] Deleting tablet eeae4369be954d9d9bad846484afb6ff
0501 13:57:42.074409 (+ 68us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.116157 (+ 41748us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.134346 (+ 18189us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.149767 (+ 15421us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255480}
I20260501 13:57:42.161319 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 220) took 1280 ms. Trace:
I20260501 13:57:42.161469 1883 tablet_bootstrap.cc:492] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:42.161892 1883 ts_tablet_manager.cc:1403] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.213s user 0.014s sys 0.027s
I20260501 13:57:42.161638 2093 rpcz_store.cc:276] 0501 13:57:40.880729 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.880857 (+ 128us) service_pool.cc:224] Handling call
0501 13:57:42.160975 (+1280118us) ts_tablet_manager.cc:1151] Deleting tablet 1bb7fdcad3a54317b43bfd58f08e89d0
0501 13:57:42.161018 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.161262 (+ 244us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1279613}
I20260501 13:57:42.162770 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 221) took 1275 ms. Trace:
I20260501 13:57:42.163079 2093 rpcz_store.cc:276] 0501 13:57:40.887011 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.887147 (+ 136us) service_pool.cc:224] Handling call
0501 13:57:42.162419 (+1275272us) ts_tablet_manager.cc:1151] Deleting tablet 1ab056da506f454984a07026341c534b
0501 13:57:42.162486 (+ 67us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.162716 (+ 230us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1274765}
I20260501 13:57:42.163970 2093 tablet_replica.cc:333] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:42.164707 2093 raft_consensus.cc:2243] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:42.165139 2093 raft_consensus.cc:2272] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:42.169574 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_fb33a28f96da45c29f6e68281ec90b8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:42.171399 1881 raft_consensus.cc:359] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.172013 1881 raft_consensus.cc:385] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:42.172250 1881 raft_consensus.cc:740] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:42.176206 1882 tablet_bootstrap.cc:654] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:42.178669 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb33a28f96da45c29f6e68281ec90b8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:42.180299 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_31a3f0b22f09450faa99203169288584"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:42.184599 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31a3f0b22f09450faa99203169288584 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:42.161885 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet eeae4369be954d9d9bad846484afb6ff (table default.Table_6452fc6a9103447c9239a13e344f0c5a [id=ecd108cb5a5544c590e904bc9185a0d0]) successfully deleted
W20260501 13:57:42.188690 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 1bb7fdcad3a54317b43bfd58f08e89d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bb7fdcad3a54317b43bfd58f08e89d0 already in progress: creating tablet
I20260501 13:57:42.189759 2047 ts_tablet_manager.cc:1929] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260501 13:57:42.190649 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 1ab056da506f454984a07026341c534b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ab056da506f454984a07026341c534b already in progress: creating tablet
I20260501 13:57:42.191444 2370 raft_consensus.cc:493] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f90b7ff170b4eadb27abc9ef866ffcf)
I20260501 13:57:42.191689 2093 ts_tablet_manager.cc:1916] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:42.191890 2370 raft_consensus.cc:515] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.192296 1883 raft_consensus.cc:359] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.192881 1883 raft_consensus.cc:385] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:42.193125 1883 raft_consensus.cc:740] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:42.193676 1883 consensus_queue.cc:260] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.194118 2370 leader_election.cc:290] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:42.195538 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 210) took 2241 ms. Trace:
I20260501 13:57:42.195840 1899 rpcz_store.cc:276] 0501 13:57:39.954020 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:39.954120 (+ 100us) service_pool.cc:224] Handling call
0501 13:57:42.037357 (+2083237us) ts_tablet_manager.cc:1151] Deleting tablet 51ba7b8a7f4d4ec6ba4af775596d3b59
0501 13:57:42.038275 (+ 918us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.108692 (+ 70417us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.158111 (+ 49419us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.195483 (+ 37372us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3389,"tablet-delete.queue_time_us":2082694}
I20260501 13:57:42.197031 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 211) took 2195 ms. Trace:
I20260501 13:57:42.197306 1899 rpcz_store.cc:276] 0501 13:57:40.001792 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.001897 (+ 105us) service_pool.cc:224] Handling call
0501 13:57:42.196721 (+2194824us) ts_tablet_manager.cc:1151] Deleting tablet b125d1a190f947a5b2ef942f04408c51
0501 13:57:42.196761 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.196983 (+ 222us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2194197}
I20260501 13:57:42.198179 1899 tablet_replica.cc:333] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:42.198662 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 (table default.Table_7b8be07c742f48d0af3422dfea143dd1 [id=39aeda29815d474196136813b5d811e7]) successfully deleted
I20260501 13:57:42.202970 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6650472ec4a449c48c7957a9bbec88e3" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:42.204849 1882 tablet_bootstrap.cc:492] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:42.205291 1882 ts_tablet_manager.cc:1403] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.002s
I20260501 13:57:42.205358 1899 raft_consensus.cc:2243] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:42.205855 1899 raft_consensus.cc:2272] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:42.206703 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet b125d1a190f947a5b2ef942f04408c51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b125d1a190f947a5b2ef942f04408c51 already in progress: creating tablet
I20260501 13:57:42.190264 2047 log.cc:1199] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/863a10cfd14c4694b7c6141c0851167a
I20260501 13:57:42.207630 1899 ts_tablet_manager.cc:1916] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:42.207945 1778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6650472ec4a449c48c7957a9bbec88e3" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:42.208684 1882 raft_consensus.cc:359] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.209292 1882 raft_consensus.cc:385] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:42.209540 1882 raft_consensus.cc:740] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:42.209491 1418 leader_election.cc:304] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:42.211032 2425 raft_consensus.cc:2749] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:57:42.212203 1883 ts_tablet_manager.cc:1434] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20260501 13:57:42.213104 1883 tablet_bootstrap.cc:492] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:42.213073 2047 ts_tablet_manager.cc:1950] T 863a10cfd14c4694b7c6141c0851167a P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:42.214425 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 863a10cfd14c4694b7c6141c0851167a (table default.Table_95c99739e2574437b16501060ce816af [id=a6953bc50bab4e7f92adfee0ef034465]) successfully deleted
I20260501 13:57:42.216244 2047 tablet_replica.cc:333] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:42.216959 2047 raft_consensus.cc:2243] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:42.217397 2047 raft_consensus.cc:2272] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:42.220628 1881 consensus_queue.cc:260] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.228334 1882 consensus_queue.cc:260] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.230423 1882 ts_tablet_manager.cc:1434] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.025s user 0.004s sys 0.002s
I20260501 13:57:42.233137 1881 ts_tablet_manager.cc:1434] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.118s user 0.006s sys 0.000s
I20260501 13:57:42.244251 1882 tablet_bootstrap.cc:492] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:42.257256 1601 tablet_service.cc:1511] Processing CreateTablet for tablet d6fd075de0b44aeab672a4ec66e8125a (DEFAULT_TABLE table=default.Table_1d3fc1cda88b46c0814a5c4edaa64340 [id=565f3bdd30914fae9bf6ea09dce58a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.258976 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fd075de0b44aeab672a4ec66e8125a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.261026 1124 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (table default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf]). Expected version 1 got 0
I20260501 13:57:42.262426 1130 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_8876cd859ccb476ea8bec2d65b8d0d60 [id=412abad469fd41d3ba02a3b0926e7578] alter complete (version 1)
I20260501 13:57:42.263006 1763 tablet_service.cc:1511] Processing CreateTablet for tablet d6fd075de0b44aeab672a4ec66e8125a (DEFAULT_TABLE table=default.Table_1d3fc1cda88b46c0814a5c4edaa64340 [id=565f3bdd30914fae9bf6ea09dce58a40]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:42.263301 1130 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.130:35289 (request call id 68) took 773 ms (client timeout 1000 ms). Trace:
I20260501 13:57:42.263967 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fd075de0b44aeab672a4ec66e8125a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:42.263589 1130 rpcz_store.cc:269] 0501 13:57:41.489624 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:41.489860 (+ 236us) service_pool.cc:224] Handling call
0501 13:57:41.530541 (+ 40681us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663025470 00007f56630c431a 00007f5664dd846c 00007f5664ed23ee 00007f5662ad94a4 00007f5662ad9432 00007f5662ad9361 00007f5662ad92dc 00007f5662ad8592 00007f5660daebcc 00007f5660dae006
0501 13:57:42.263244 (+732703us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:41.648353 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:41.648753 (+ 400us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:41.648777 (+ 24us) write_op.cc:435] Acquired schema lock
0501 13:57:41.648790 (+ 13us) tablet.cc:662] Decoding operations
0501 13:57:41.649030 (+ 240us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:41.649064 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:41.649085 (+ 21us) tablet.cc:685] Acquiring locks for 1 operations
0501 13:57:41.649222 (+ 137us) tablet.cc:701] Row locks acquired
0501 13:57:41.649235 (+ 13us) write_op.cc:260] PREPARE: finished
0501 13:57:41.649336 (+ 101us) write_op.cc:270] Start()
0501 13:57:41.649414 (+ 78us) write_op.cc:276] Timestamp: P: 1777643861649319 usec, L: 0
0501 13:57:41.649428 (+ 14us) op_driver.cc:348] REPLICATION: starting
0501 13:57:41.670419 (+ 20991us) log.cc:844] Serialized 356 byte log entry
0501 13:57:41.856168 (+185749us) op_driver.cc:464] REPLICATION: finished
0501 13:57:41.856624 (+ 456us) write_op.cc:301] APPLY: starting
0501 13:57:41.856686 (+ 62us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:41.856767 (+ 81us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:41.856865 (+ 98us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:41.857016 (+ 151us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:41.857143 (+ 127us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:57:41.857181 (+ 38us) write_op.cc:312] APPLY: finished
0501 13:57:41.857345 (+ 164us) log.cc:844] Serialized 23 byte log entry
0501 13:57:41.857957 (+ 612us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:41.858026 (+ 69us) write_op.cc:454] Released schema lock
0501 13:57:41.858055 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:41.858384 (+ 329us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:41.858406 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":86726400,"child_traces":[["op",{"apply.queue_time_us":317,"apply.run_cpu_time_us":1187,"apply.run_wall_time_us":1725,"num_ops":1,"prepare.queue_time_us":113537,"prepare.run_cpu_time_us":1850,"prepare.run_wall_time_us":23654,"raft.queue_time_us":1052,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":13,"replication_time_us":206645}]]}
I20260501 13:57:42.269002 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 812b31a247a345ccb6b33e8f87ca8065 (DEFAULT_TABLE table=default.Table_f9da71780b99473393cc5d38e2d4ca2d [id=070f410444a944189992a11d60b381c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.269984 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812b31a247a345ccb6b33e8f87ca8065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.267145 1610 tablet_service.cc:1511] Processing CreateTablet for tablet 812b31a247a345ccb6b33e8f87ca8065 (DEFAULT_TABLE table=default.Table_f9da71780b99473393cc5d38e2d4ca2d [id=070f410444a944189992a11d60b381c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.272624 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812b31a247a345ccb6b33e8f87ca8065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.273777 2409 raft_consensus.cc:493] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:42.274194 2409 raft_consensus.cc:515] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:42.275552 1124 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.131:42895 (request call id 66) took 850 ms (client timeout 1000 ms). Trace:
I20260501 13:57:42.276010 2409 leader_election.cc:290] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:42.275884 1124 rpcz_store.cc:269] 0501 13:57:41.425622 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:41.428203 (+ 2581us) service_pool.cc:224] Handling call
0501 13:57:41.603085 (+174882us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663025470 00007f56630c431a 00007f5664dd846c 00007f5664ed23ee 00007f5662ad94a4 00007f5662ad9432 00007f5662ad9361 00007f5662ad92dc 00007f5662ad8592 00007f5660daebcc 00007f5660dae006
0501 13:57:42.275488 (+672403us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2670,"reported_tablets":2,"spinlock_wait_cycles":397073152}
W20260501 13:57:42.279469 1563 leader_election.cc:343] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: a9672aab85c64d548a4f6583a1e9af20
I20260501 13:57:42.286100 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 2b7e55c79aa74ee6b164758daf6325fa (DEFAULT_TABLE table=default.Table_58189c0095284474a78cf7aa10654434 [id=5ee82272f6f441488920aa56b105e29d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.287182 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b7e55c79aa74ee6b164758daf6325fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.289036 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 39c3f2884ce049a887d57929685a3629 (DEFAULT_TABLE table=default.Table_bcf66326bed34b2ba3bba85f08f2573b [id=f73aec97182348f592834f1eb46a5280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.289990 1882 tablet_bootstrap.cc:654] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:42.290004 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39c3f2884ce049a887d57929685a3629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.243636 2047 ts_tablet_manager.cc:1916] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:42.254308 1465 tablet_service.cc:1511] Processing CreateTablet for tablet d6fd075de0b44aeab672a4ec66e8125a (DEFAULT_TABLE table=default.Table_1d3fc1cda88b46c0814a5c4edaa64340 [id=565f3bdd30914fae9bf6ea09dce58a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.291332 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fd075de0b44aeab672a4ec66e8125a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.291548 1755 tablet_service.cc:1511] Processing CreateTablet for tablet 2b7e55c79aa74ee6b164758daf6325fa (DEFAULT_TABLE table=default.Table_58189c0095284474a78cf7aa10654434 [id=5ee82272f6f441488920aa56b105e29d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.292498 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b7e55c79aa74ee6b164758daf6325fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.263706 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 812b31a247a345ccb6b33e8f87ca8065 (DEFAULT_TABLE table=default.Table_f9da71780b99473393cc5d38e2d4ca2d [id=070f410444a944189992a11d60b381c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.293280 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812b31a247a345ccb6b33e8f87ca8065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.270160 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 2b7e55c79aa74ee6b164758daf6325fa (DEFAULT_TABLE table=default.Table_58189c0095284474a78cf7aa10654434 [id=5ee82272f6f441488920aa56b105e29d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.278510 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9672aab85c64d548a4f6583a1e9af20" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:42.295046 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b7e55c79aa74ee6b164758daf6325fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.295516 1881 tablet_bootstrap.cc:492] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:42.296576 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 39c3f2884ce049a887d57929685a3629 (DEFAULT_TABLE table=default.Table_bcf66326bed34b2ba3bba85f08f2573b [id=f73aec97182348f592834f1eb46a5280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.297534 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39c3f2884ce049a887d57929685a3629. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:42.300037 1564 leader_election.cc:343] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.300467 1564 leader_election.cc:304] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.301584 2409 raft_consensus.cc:2749] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:42.302160 1881 tablet_bootstrap.cc:654] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:42.313977 1129 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.129:53385 (request call id 65) took 891 ms (client timeout 1000 ms). Trace:
W20260501 13:57:42.314368 1129 rpcz_store.cc:269] 0501 13:57:41.422132 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:41.422428 (+ 296us) service_pool.cc:224] Handling call
0501 13:57:41.444714 (+ 22286us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663025470 00007f56630c431a 00007f5664dd846c 00007f5664ed23ee 00007f5662ad94a4 00007f5662ad9432 00007f5662ad9361 00007f5662ad92dc 00007f5662ad8592 00007f5660daebcc 00007f5660dae006
0501 13:57:42.313905 (+869191us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":47497088}
I20260501 13:57:42.315805 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a0d04810f2d465ebe2964797ea30538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:42.316499 2093 ts_tablet_manager.cc:1929] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:42.316864 2093 log.cc:1199] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/4f2448ee6bf14bbc8ec0e1d59ccb7b90
I20260501 13:57:42.321805 2093 ts_tablet_manager.cc:1950] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:42.324182 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 (table default.Table_041fce4b819d4a4c80c58137b4213234 [id=797990ef44df4a5685fa58bd4c3547bd]) successfully deleted
I20260501 13:57:42.328460 1455 tablet_service.cc:1511] Processing CreateTablet for tablet 2c22c2f8e55a4be99a8c6ff32bace441 (DEFAULT_TABLE table=default.Table_de443751750240b6a298d8aca41e6b56 [id=d1cfd215cd17431ba79e6d2645eea53f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.329001 2093 tablet_replica.cc:333] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:42.330385 1602 tablet_service.cc:1511] Processing CreateTablet for tablet 2c22c2f8e55a4be99a8c6ff32bace441 (DEFAULT_TABLE table=default.Table_de443751750240b6a298d8aca41e6b56 [id=d1cfd215cd17431ba79e6d2645eea53f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.331382 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c22c2f8e55a4be99a8c6ff32bace441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.335258 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 39c3f2884ce049a887d57929685a3629 (DEFAULT_TABLE table=default.Table_bcf66326bed34b2ba3bba85f08f2573b [id=f73aec97182348f592834f1eb46a5280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.336205 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39c3f2884ce049a887d57929685a3629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.339385 2423 raft_consensus.cc:493] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:42.339805 2423 raft_consensus.cc:515] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.341660 2423 leader_election.cc:290] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:42.345814 1418 leader_election.cc:343] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 1ba63c153ef04db891cae46a71439455
I20260501 13:57:42.352171 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 2c22c2f8e55a4be99a8c6ff32bace441 (DEFAULT_TABLE table=default.Table_de443751750240b6a298d8aca41e6b56 [id=d1cfd215cd17431ba79e6d2645eea53f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.353695 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ba63c153ef04db891cae46a71439455" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:42.354247 1629 raft_consensus.cc:2468] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:42.354661 2093 raft_consensus.cc:2243] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:42.356978 1421 leader_election.cc:304] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:42.355114 2093 raft_consensus.cc:2272] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:42.359545 2423 raft_consensus.cc:2804] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:42.359953 2423 raft_consensus.cc:493] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:42.360589 2423 raft_consensus.cc:3060] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:42.362375 1464 tablet_service.cc:1511] Processing CreateTablet for tablet d4d93d0fc9e14d7abc126abbb22787e9 (DEFAULT_TABLE table=default.Table_645fbdfab04b4def8a0d842bd01c2ade [id=5a00b01af545446ea005b3d7ccdd9e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.363544 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d93d0fc9e14d7abc126abbb22787e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.363502 1125 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 15a1f80cf7d24974b00c6cca7ae5783e
I20260501 13:57:42.364686 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4e1547b16dcc4d878bcc141802b827dc" } new_table_name: "default.Table_33be6d10669d46efad19a4393e18553d" modify_external_catalogs: true
I20260501 13:57:42.365878 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet 15a1f80cf7d24974b00c6cca7ae5783e with delete_type TABLET_DATA_DELETED (replaced by tablet 6399ac6a0d044ad2aed95ba30e6659b8) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:42.366379 1608 tablet_service.cc:1558] Processing DeleteTablet for tablet 15a1f80cf7d24974b00c6cca7ae5783e with delete_type TABLET_DATA_DELETED (replaced by tablet 6399ac6a0d044ad2aed95ba30e6659b8) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:42.368438 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet 15a1f80cf7d24974b00c6cca7ae5783e with delete_type TABLET_DATA_DELETED (replaced by tablet 6399ac6a0d044ad2aed95ba30e6659b8) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:42.368824 1608 tablet_service.cc:1511] Processing CreateTablet for tablet d4d93d0fc9e14d7abc126abbb22787e9 (DEFAULT_TABLE table=default.Table_645fbdfab04b4def8a0d842bd01c2ade [id=5a00b01af545446ea005b3d7ccdd9e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.369778 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d93d0fc9e14d7abc126abbb22787e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.369638 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_d132a7668b34479887548c901967b36c" } new_table_name: "default.Table_1da5624be9fd43498a32173c9025a446" modify_external_catalogs: true
I20260501 13:57:42.369725 1882 tablet_bootstrap.cc:492] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:42.370288 1882 ts_tablet_manager.cc:1403] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.126s user 0.004s sys 0.006s
I20260501 13:57:42.375715 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c22c2f8e55a4be99a8c6ff32bace441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.378822 2093 ts_tablet_manager.cc:1916] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:42.384157 2047 ts_tablet_manager.cc:1929] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:42.384534 2047 log.cc:1199] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/eeae4369be954d9d9bad846484afb6ff
I20260501 13:57:42.385131 2047 ts_tablet_manager.cc:1950] T eeae4369be954d9d9bad846484afb6ff P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:42.381532 1763 tablet_service.cc:1511] Processing CreateTablet for tablet d4d93d0fc9e14d7abc126abbb22787e9 (DEFAULT_TABLE table=default.Table_645fbdfab04b4def8a0d842bd01c2ade [id=5a00b01af545446ea005b3d7ccdd9e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.386479 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d93d0fc9e14d7abc126abbb22787e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.364753 1125 catalog_manager.cc:6741] ReplaceTablet: tablet 15a1f80cf7d24974b00c6cca7ae5783e deleted and replaced by tablet 6399ac6a0d044ad2aed95ba30e6659b8
I20260501 13:57:42.393132 1123 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 6650472ec4a449c48c7957a9bbec88e3 (table default.Table_0d92e7b95c884bd5970e291f4189b0d9 [id=774a7eb90ba449b6aef225d819678439]). Expected version 1 got 0
I20260501 13:57:42.377645 1882 raft_consensus.cc:359] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.394516 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet eeae4369be954d9d9bad846484afb6ff (table default.Table_6452fc6a9103447c9239a13e344f0c5a [id=ecd108cb5a5544c590e904bc9185a0d0]) successfully deleted
I20260501 13:57:42.394719 1882 raft_consensus.cc:385] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:42.393968 1123 catalog_manager.cc:5671] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:42.395011 1882 raft_consensus.cc:740] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:42.395772 1123 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 406e1af8b7d9488f88697206daa87c05 (table default.Table_759f3009542147de94c73784e1698246 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]). Expected version 1 got 0
I20260501 13:57:42.395591 1882 consensus_queue.cc:260] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.397017 1123 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715]). Expected version 1 got 0
I20260501 13:57:42.397550 1882 ts_tablet_manager.cc:1434] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.027s user 0.003s sys 0.000s
I20260501 13:57:42.402834 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c22c2f8e55a4be99a8c6ff32bace441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.403589 1757 tablet_service.cc:1511] Processing CreateTablet for tablet ee76efc1e2204502b61bc5c7892ed856 (DEFAULT_TABLE table=default.Table_bc194cec5138404eb7ba98e3bfb0ea5d [id=ebc44f52a33846f3b66b4c9d498f9651]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.404645 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee76efc1e2204502b61bc5c7892ed856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.406662 1883 tablet_bootstrap.cc:700] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent opening tablet: real 0.189s user 0.003s sys 0.001s
I20260501 13:57:42.407241 1883 tablet_bootstrap.cc:654] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:42.407176 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 90ea683130884dc4b60e9ccadf6166c0 (DEFAULT_TABLE table=default.Table_7c25358c17d94467828a54e6fd12be5a [id=ede720f4a1504d91880d4408ff286b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.410380 1753 tablet_service.cc:1511] Processing CreateTablet for tablet 90ea683130884dc4b60e9ccadf6166c0 (DEFAULT_TABLE table=default.Table_7c25358c17d94467828a54e6fd12be5a [id=ede720f4a1504d91880d4408ff286b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.411384 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90ea683130884dc4b60e9ccadf6166c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.412843 1603 tablet_service.cc:1511] Processing CreateTablet for tablet 3c4e7af3f6674e4ebb9bae5f53d83c93 (DEFAULT_TABLE table=default.Table_7f0f65b600aa434cbe776589a4ec2855 [id=61b7774aaf8e4ef5a2faf998b09105be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.422143 2424 raft_consensus.cc:493] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:42.422622 2424 raft_consensus.cc:515] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.422905 1123 catalog_manager.cc:5671] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:42.424243 1123 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 96e937b3a0b5451e8eaf4d5c47c17620 (table default.Table_b29786a52abd4fad9a2337d7c76fbbc4 [id=05998ead56ed454dab9a71ef2ac9cb58]). Expected version 1 got 0
I20260501 13:57:42.424376 2424 leader_election.cc:290] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:42.389374 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_a308b6848f114c1ead6ecf2798c61d37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:42.426982 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 4eaf33dce3564d38a36910e047474228 (DEFAULT_TABLE table=default.Table_2c4ab34d8a9b4072b1cc2cb7c5ca3203 [id=5edbe4fbb4584ad499af039fe8a793ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.427793 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7ff8525575ef446aad5a61795d4018a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:42.428130 1752 tablet_service.cc:1511] Processing CreateTablet for tablet 3c4e7af3f6674e4ebb9bae5f53d83c93 (DEFAULT_TABLE table=default.Table_7f0f65b600aa434cbe776589a4ec2855 [id=61b7774aaf8e4ef5a2faf998b09105be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.428228 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "569a0240b1274d9bb830addbaf1ec343" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:42.429100 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4e7af3f6674e4ebb9bae5f53d83c93. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:42.429250 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ff8525575ef446aad5a61795d4018a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:42.430254 1717 leader_election.cc:343] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.431267 2423 raft_consensus.cc:515] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.402580 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7fdfc794bcaf49488caa439514624cec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:42.433398 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fdfc794bcaf49488caa439514624cec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:42.437683 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a308b6848f114c1ead6ecf2798c61d37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:42.448354 1127 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 38c3eb388d474d718554775d65f58d78
W20260501 13:57:42.398921 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 8dbb619863dd405ab20ead6bbebf4d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dbb619863dd405ab20ead6bbebf4d69 already in progress: creating tablet
W20260501 13:57:42.449133 1117 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 50251.4us
I20260501 13:57:42.450647 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 38c3eb388d474d718554775d65f58d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:42.450699 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 38c3eb388d474d718554775d65f58d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:42.451916 1456 tablet_service.cc:1511] Processing CreateTablet for tablet ee76efc1e2204502b61bc5c7892ed856 (DEFAULT_TABLE table=default.Table_bc194cec5138404eb7ba98e3bfb0ea5d [id=ebc44f52a33846f3b66b4c9d498f9651]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.454774 1754 tablet_service.cc:1511] Processing CreateTablet for tablet 4eaf33dce3564d38a36910e047474228 (DEFAULT_TABLE table=default.Table_2c4ab34d8a9b4072b1cc2cb7c5ca3203 [id=5edbe4fbb4584ad499af039fe8a793ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.455295 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_9bd7fbc81aa64971ab1296f32b26179f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:42.455883 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eaf33dce3564d38a36910e047474228. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:42.456802 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bd7fbc81aa64971ab1296f32b26179f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:42.460024 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 90ea683130884dc4b60e9ccadf6166c0 (DEFAULT_TABLE table=default.Table_7c25358c17d94467828a54e6fd12be5a [id=ede720f4a1504d91880d4408ff286b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.467182 1460 tablet_service.cc:1511] Processing CreateTablet for tablet 3c4e7af3f6674e4ebb9bae5f53d83c93 (DEFAULT_TABLE table=default.Table_7f0f65b600aa434cbe776589a4ec2855 [id=61b7774aaf8e4ef5a2faf998b09105be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.467792 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "569a0240b1274d9bb830addbaf1ec343" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:42.469298 1719 leader_election.cc:343] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.469699 1719 leader_election.cc:304] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.470073 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ba63c153ef04db891cae46a71439455" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:42.470796 2424 raft_consensus.cc:2749] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:42.471330 1421 leader_election.cc:343] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: Tablet not RUNNING: STOPPED
I20260501 13:57:42.471810 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eaf33dce3564d38a36910e047474228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.475818 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 30a1ef8502c648ce8da3905e4dff4c0c (DEFAULT_TABLE table=default.Table_33fb06bbd21c4209919e7b4711c86e48 [id=49870caf0d0d49059424859f3a594352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.476531 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 4eaf33dce3564d38a36910e047474228 (DEFAULT_TABLE table=default.Table_2c4ab34d8a9b4072b1cc2cb7c5ca3203 [id=5edbe4fbb4584ad499af039fe8a793ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.476814 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a1ef8502c648ce8da3905e4dff4c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.480329 2093 ts_tablet_manager.cc:1929] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:42.481168 2093 log.cc:1199] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/1ba63c153ef04db891cae46a71439455
I20260501 13:57:42.481786 2093 ts_tablet_manager.cc:1950] T 1ba63c153ef04db891cae46a71439455 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:42.482723 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet 38c3eb388d474d718554775d65f58d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:42.483515 1096 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 47600us
I20260501 13:57:42.483569 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 239) took 1049 ms. Trace:
I20260501 13:57:42.485265 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 30a1ef8502c648ce8da3905e4dff4c0c (DEFAULT_TABLE table=default.Table_33fb06bbd21c4209919e7b4711c86e48 [id=49870caf0d0d49059424859f3a594352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.486761 2409 raft_consensus.cc:493] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:42.492841 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee76efc1e2204502b61bc5c7892ed856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.528630 2409 raft_consensus.cc:515] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.530489 2409 leader_election.cc:290] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:42.496387 2302 raft_consensus.cc:493] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:42.531028 2302 raft_consensus.cc:515] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.532259 2047 tablet_replica.cc:333] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:42.532979 2047 raft_consensus.cc:2243] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:42.533349 2302 leader_election.cc:290] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:42.533582 2047 pending_rounds.cc:70] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: Trying to abort 1 pending ops.
I20260501 13:57:42.533777 2047 pending_rounds.cc:77] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229258834944000 op_type: NO_OP noop_request { }
I20260501 13:57:42.534080 2047 raft_consensus.cc:2889] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:42.534770 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a1ef8502c648ce8da3905e4dff4c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.535938 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "406e1af8b7d9488f88697206daa87c05" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:42.536909 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96e937b3a0b5451e8eaf4d5c47c17620" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:42.540997 1564 leader_election.cc:343] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:42.541610 1564 leader_election.cc:343] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.543519 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4e7af3f6674e4ebb9bae5f53d83c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.547348 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eaf33dce3564d38a36910e047474228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.505126 1464 tablet_service.cc:1511] Processing CreateTablet for tablet ea425f73649546648a1082ee75b8f116 (DEFAULT_TABLE table=default.Table_5d773086180e4cd09927c8b2fe2e650b [id=194d1268b0c54fc494bc404239aaeeaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.508209 1881 tablet_bootstrap.cc:492] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:42.548435 1881 ts_tablet_manager.cc:1403] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.253s user 0.013s sys 0.000s
I20260501 13:57:42.549571 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90ea683130884dc4b60e9ccadf6166c0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:42.521086 1418 leader_election.cc:343] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 1ba63c153ef04db891cae46a71439455
I20260501 13:57:42.521486 2423 leader_election.cc:290] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:42.534359 2047 raft_consensus.cc:2272] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:42.550668 1418 leader_election.cc:304] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:42.551672 2423 raft_consensus.cc:2749] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260501 13:57:42.551838 1881 raft_consensus.cc:359] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.552425 1881 raft_consensus.cc:385] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:42.552651 1881 raft_consensus.cc:740] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:42.553182 1881 consensus_queue.cc:260] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.553648 2047 ts_tablet_manager.cc:1916] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:42.556541 1881 ts_tablet_manager.cc:1434] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260501 13:57:42.557410 1881 tablet_bootstrap.cc:492] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:42.499197 1601 tablet_service.cc:1511] Processing CreateTablet for tablet ee76efc1e2204502b61bc5c7892ed856 (DEFAULT_TABLE table=default.Table_bc194cec5138404eb7ba98e3bfb0ea5d [id=ebc44f52a33846f3b66b4c9d498f9651]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.508946 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4e7af3f6674e4ebb9bae5f53d83c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.511037 1609 tablet_service.cc:1511] Processing CreateTablet for tablet ea425f73649546648a1082ee75b8f116 (DEFAULT_TABLE table=default.Table_5d773086180e4cd09927c8b2fe2e650b [id=194d1268b0c54fc494bc404239aaeeaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.515046 1882 tablet_bootstrap.cc:492] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:42.525617 2344 raft_consensus.cc:493] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:42.567070 2344 raft_consensus.cc:515] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.568903 2344 leader_election.cc:290] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:42.567071 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90ea683130884dc4b60e9ccadf6166c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.570453 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9abb39c8b44d4b4c8b69751a4cdd6668" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:42.570894 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea425f73649546648a1082ee75b8f116. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:42.571936 1563 leader_election.cc:343] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 9abb39c8b44d4b4c8b69751a4cdd6668
I20260501 13:57:42.574604 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea425f73649546648a1082ee75b8f116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.575896 1882 tablet_bootstrap.cc:654] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:42.579399 1564 leader_election.cc:343] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.579792 1564 leader_election.cc:304] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.580679 2344 raft_consensus.cc:2749] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:42.590684 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee76efc1e2204502b61bc5c7892ed856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.599804 1882 tablet_bootstrap.cc:492] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:42.600205 1882 ts_tablet_manager.cc:1403] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.005s
I20260501 13:57:42.497840 1752 tablet_service.cc:1511] Processing CreateTablet for tablet 30a1ef8502c648ce8da3905e4dff4c0c (DEFAULT_TABLE table=default.Table_33fb06bbd21c4209919e7b4711c86e48 [id=49870caf0d0d49059424859f3a594352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.498517 1883 tablet_bootstrap.cc:492] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:42.600893 1883 ts_tablet_manager.cc:1403] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.388s user 0.009s sys 0.002s
I20260501 13:57:42.601279 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a1ef8502c648ce8da3905e4dff4c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.602447 1882 raft_consensus.cc:359] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.603255 1883 raft_consensus.cc:359] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.603856 1883 raft_consensus.cc:385] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:42.604111 1883 raft_consensus.cc:740] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:42.604660 1883 consensus_queue.cc:260] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.607278 1881 tablet_bootstrap.cc:654] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:42.594545 2093 rpcz_store.cc:276] 0501 13:57:41.434231 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:41.438009 (+ 3778us) service_pool.cc:224] Handling call
0501 13:57:42.328859 (+890850us) ts_tablet_manager.cc:1151] Deleting tablet 1ba63c153ef04db891cae46a71439455
0501 13:57:42.328900 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.407792 (+ 78892us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.480218 (+ 72426us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.483513 (+ 3295us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":890213}
I20260501 13:57:42.609009 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 1ba63c153ef04db891cae46a71439455 (table default.Table_66a4e496efa34deaa6c793a210a510ff [id=e8b19350d702499a82f41a6cce14dbd5]) successfully deleted
I20260501 13:57:42.609977 2093 tablet_replica.cc:333] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:42.514609 1755 tablet_service.cc:1511] Processing CreateTablet for tablet ea425f73649546648a1082ee75b8f116 (DEFAULT_TABLE table=default.Table_5d773086180e4cd09927c8b2fe2e650b [id=194d1268b0c54fc494bc404239aaeeaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.538547 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96e937b3a0b5451e8eaf4d5c47c17620" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:42.610787 2093 raft_consensus.cc:2243] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:42.610834 1782 raft_consensus.cc:2468] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:42.611172 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea425f73649546648a1082ee75b8f116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.539680 1778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "406e1af8b7d9488f88697206daa87c05" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:42.611639 1778 raft_consensus.cc:2468] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:42.611964 1563 leader_election.cc:304] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.565827 2424 raft_consensus.cc:493] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:42.612576 2424 raft_consensus.cc:515] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.613080 1563 leader_election.cc:304] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.613988 2302 raft_consensus.cc:2804] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:42.614271 2302 raft_consensus.cc:493] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:42.614610 2302 raft_consensus.cc:3060] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:42.616088 2424 leader_election.cc:290] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:42.617322 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69626ed095bd4df7859563c176dafffc" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:42.603061 1882 raft_consensus.cc:385] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:42.618239 1882 raft_consensus.cc:740] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:42.619055 2344 raft_consensus.cc:2804] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:42.618795 1882 consensus_queue.cc:260] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.619376 2344 raft_consensus.cc:493] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:42.619694 2344 raft_consensus.cc:3060] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:42.622895 1719 leader_election.cc:343] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:42.639081 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet daca20eeafb141f2bf8b4233e0c8ea3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daca20eeafb141f2bf8b4233e0c8ea3b already in progress: creating tablet
W20260501 13:57:42.639674 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet f52ea8fb8ae0427d804001aed42cd67c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f52ea8fb8ae0427d804001aed42cd67c already in progress: creating tablet
I20260501 13:57:42.642906 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 6ea701a51a9440f283a6934ef5771979 (DEFAULT_TABLE table=default.Table_e3ff5a5e561a442d9af7f6c8317bd75b [id=a820b5b40cc04093a36c6602d01fd5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.643873 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea701a51a9440f283a6934ef5771979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.644116 1757 tablet_service.cc:1511] Processing CreateTablet for tablet 6ea701a51a9440f283a6934ef5771979 (DEFAULT_TABLE table=default.Table_e3ff5a5e561a442d9af7f6c8317bd75b [id=a820b5b40cc04093a36c6602d01fd5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.611195 2093 raft_consensus.cc:2272] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:42.645117 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea701a51a9440f283a6934ef5771979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.645241 2344 raft_consensus.cc:515] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.625380 2302 raft_consensus.cc:515] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.648480 1604 tablet_service.cc:1511] Processing CreateTablet for tablet df3d5bbf0ecd475494f74ecc29d80083 (DEFAULT_TABLE table=default.Table_f12194a7039e403eb237837a2e2653e9 [id=ef709e4b7fab4ff3a9e35815f43d681e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.649384 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3d5bbf0ecd475494f74ecc29d80083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.654772 2302 leader_election.cc:290] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:42.655728 1778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96e937b3a0b5451e8eaf4d5c47c17620" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:42.656247 1778 raft_consensus.cc:3060] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:42.598276 1899 ts_tablet_manager.cc:1929] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:57:42.663239 1899 log.cc:1199] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/b8d0f6d1f38b428291dba22015ac6959
I20260501 13:57:42.664214 1899 ts_tablet_manager.cc:1950] T b8d0f6d1f38b428291dba22015ac6959 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:42.664584 1882 ts_tablet_manager.cc:1434] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.064s user 0.006s sys 0.000s
I20260501 13:57:42.665807 1882 tablet_bootstrap.cc:492] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:42.662358 1778 raft_consensus.cc:2468] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:42.667424 1563 leader_election.cc:304] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
W20260501 13:57:42.669945 1564 consensus_peers.cc:597] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:42.672380 2344 leader_election.cc:290] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:42.672842 2344 raft_consensus.cc:2804] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:42.646301 2093 ts_tablet_manager.cc:1916] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:42.678659 1778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "406e1af8b7d9488f88697206daa87c05" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:42.679188 1778 raft_consensus.cc:3060] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:42.681550 2344 raft_consensus.cc:697] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:42.686285 2344 consensus_queue.cc:237] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.687211 2424 raft_consensus.cc:493] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:42.687638 2424 raft_consensus.cc:515] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.690855 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 212) took 2163 ms. Trace:
I20260501 13:57:42.691390 2424 leader_election.cc:290] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:42.691164 1899 rpcz_store.cc:276] 0501 13:57:40.527667 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.527798 (+ 131us) service_pool.cc:224] Handling call
0501 13:57:42.198030 (+1670232us) ts_tablet_manager.cc:1151] Deleting tablet b8d0f6d1f38b428291dba22015ac6959
0501 13:57:42.198077 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.417995 (+219918us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.598159 (+180164us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.690799 (+ 92640us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6389,"tablet-delete.queue_time_us":1669696}
I20260501 13:57:42.692845 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33a355897309420c97b1257afaefe8d1" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:42.694142 1719 leader_election.cc:343] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.696027 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet b8d0f6d1f38b428291dba22015ac6959 (table default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba]) successfully deleted
I20260501 13:57:42.696949 1899 tablet_replica.cc:333] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:42.698576 1899 raft_consensus.cc:2243] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:42.615643 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69626ed095bd4df7859563c176dafffc" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:42.640868 2423 raft_consensus.cc:493] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:42.703434 2423 raft_consensus.cc:515] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.704229 1899 raft_consensus.cc:2272] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:42.705386 2423 leader_election.cc:290] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:42.648777 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96e937b3a0b5451e8eaf4d5c47c17620" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:42.705780 1899 ts_tablet_manager.cc:1916] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:42.705960 1484 raft_consensus.cc:3060] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:42.709539 2047 ts_tablet_manager.cc:1929] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:42.709874 2047 log.cc:1199] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/9a0d04810f2d465ebe2964797ea30538
I20260501 13:57:42.701975 2424 raft_consensus.cc:493] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:42.710565 2047 ts_tablet_manager.cc:1950] T 9a0d04810f2d465ebe2964797ea30538 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:42.710680 2424 raft_consensus.cc:515] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.667606 1455 tablet_service.cc:1511] Processing CreateTablet for tablet df3d5bbf0ecd475494f74ecc29d80083 (DEFAULT_TABLE table=default.Table_f12194a7039e403eb237837a2e2653e9 [id=ef709e4b7fab4ff3a9e35815f43d681e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.712246 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 9a0d04810f2d465ebe2964797ea30538 (table default.Table_a74b32dea54d43a2a8a1b6f2210f38d8 [id=eaa4c83258f94954af2e04eee604f6f0]) successfully deleted
I20260501 13:57:42.712620 1882 tablet_bootstrap.cc:654] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:42.671926 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "406e1af8b7d9488f88697206daa87c05" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
W20260501 13:57:42.712881 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 15a1f80cf7d24974b00c6cca7ae5783e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15a1f80cf7d24974b00c6cca7ae5783e already in progress: creating tablet
I20260501 13:57:42.715152 1778 raft_consensus.cc:2468] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:42.717088 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3d5bbf0ecd475494f74ecc29d80083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.719381 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2cf9528d5c24fc68e3dfdd93bf5c7bb" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:42.721633 1564 leader_election.cc:343] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:42.716472 1418 leader_election.cc:343] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 90a57bc58e854e469897a04b2d434d06
W20260501 13:57:42.722555 1421 leader_election.cc:343] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Not found: Tablet not found: 90a57bc58e854e469897a04b2d434d06
W20260501 13:57:42.723114 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 38c3eb388d474d718554775d65f58d78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38c3eb388d474d718554775d65f58d78 already in progress: creating tablet
I20260501 13:57:42.723287 1460 tablet_service.cc:1511] Processing CreateTablet for tablet c07b1d2c40b74a93866db5124d5ee47f (DEFAULT_TABLE table=default.Table_f0801819716e426f92844203c4a06bdc [id=068f5ac17d494d0b8639cbdc07a04597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.734692 2302 raft_consensus.cc:493] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:42.735158 2302 raft_consensus.cc:515] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.736526 1882 tablet_bootstrap.cc:492] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:42.736912 1882 ts_tablet_manager.cc:1403] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.000s
I20260501 13:57:42.737462 2302 leader_election.cc:290] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:42.739138 1563 leader_election.cc:343] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: ec4bd4cb1e70472aa09fc93afc3b1501
I20260501 13:57:42.739164 1882 raft_consensus.cc:359] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.740268 1882 raft_consensus.cc:385] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:42.740518 1882 raft_consensus.cc:740] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:42.741165 1882 consensus_queue.cc:260] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.741781 1603 tablet_service.cc:1511] Processing CreateTablet for tablet c07b1d2c40b74a93866db5124d5ee47f (DEFAULT_TABLE table=default.Table_f0801819716e426f92844203c4a06bdc [id=068f5ac17d494d0b8639cbdc07a04597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.742775 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c07b1d2c40b74a93866db5124d5ee47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.746308 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 9d38d75fcd494fc4a3420bcf15f2c70f (DEFAULT_TABLE table=default.Table_96180f484b044c0b85bb2f5d8a14d194 [id=48266956ca8b4274aea505435692daaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.747262 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d38d75fcd494fc4a3420bcf15f2c70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.690713 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33a355897309420c97b1257afaefe8d1" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:42.751084 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c07b1d2c40b74a93866db5124d5ee47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.724609 1421 leader_election.cc:304] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:42.754333 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 37f57009c05f479481b59e49062608d5 (DEFAULT_TABLE table=default.Table_bd8182bfb41b4025b826d479090f8f09 [id=2b1b103ff1ad4207b67ce49e8c6802b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.754868 1882 ts_tablet_manager.cc:1434] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260501 13:57:42.755316 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f57009c05f479481b59e49062608d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.755988 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec4bd4cb1e70472aa09fc93afc3b1501" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:42.757094 2423 raft_consensus.cc:2749] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:42.715219 1717 leader_election.cc:343] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.758945 1717 leader_election.cc:304] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.759402 1563 leader_election.cc:304] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:42.759711 1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 44536us
I20260501 13:57:42.760462 2302 raft_consensus.cc:2804] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
W20260501 13:57:42.760563 1717 leader_election.cc:343] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.760903 1717 leader_election.cc:304] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.761179 2302 raft_consensus.cc:697] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:42.703814 1883 ts_tablet_manager.cc:1434] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.103s user 0.013s sys 0.008s
I20260501 13:57:42.762285 1883 tablet_bootstrap.cc:492] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:42.761911 2302 consensus_queue.cc:237] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:42.718686 2424 leader_election.cc:290] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
W20260501 13:57:42.721529 1719 leader_election.cc:343] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:42.766697 1719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 45209.8us
I20260501 13:57:42.766752 2424 raft_consensus.cc:2749] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:42.767328 2424 raft_consensus.cc:2749] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:42.768512 1882 tablet_bootstrap.cc:492] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:42.769635 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2cf9528d5c24fc68e3dfdd93bf5c7bb" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:42.771144 1754 tablet_service.cc:1511] Processing CreateTablet for tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9 (DEFAULT_TABLE table=default.Table_22d1c2cee41e4e9dad9e52bde00f1524 [id=a978daa811864093b075583f4e6bc4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.772439 1763 tablet_service.cc:1511] Processing CreateTablet for tablet c07b1d2c40b74a93866db5124d5ee47f (DEFAULT_TABLE table=default.Table_f0801819716e426f92844203c4a06bdc [id=068f5ac17d494d0b8639cbdc07a04597]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.770205 1755 tablet_service.cc:1511] Processing CreateTablet for tablet 9d38d75fcd494fc4a3420bcf15f2c70f (DEFAULT_TABLE table=default.Table_96180f484b044c0b85bb2f5d8a14d194 [id=48266956ca8b4274aea505435692daaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.773697 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.773264 1883 tablet_bootstrap.cc:654] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:42.722293 1756 tablet_service.cc:1511] Processing CreateTablet for tablet df3d5bbf0ecd475494f74ecc29d80083 (DEFAULT_TABLE table=default.Table_f12194a7039e403eb237837a2e2653e9 [id=ef709e4b7fab4ff3a9e35815f43d681e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:42.780478 1564 leader_election.cc:343] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:42.783984 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d38d75fcd494fc4a3420bcf15f2c70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.740842 1899 ts_tablet_manager.cc:1929] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:42.784986 1899 log.cc:1199] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/0c736959df334aee857dbaae0f7a1e1a
I20260501 13:57:42.785553 1899 ts_tablet_manager.cc:1950] T 0c736959df334aee857dbaae0f7a1e1a P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:42.787456 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c07b1d2c40b74a93866db5124d5ee47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.788077 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df3d5bbf0ecd475494f74ecc29d80083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.802660 1564 leader_election.cc:304] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.802815 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 213) took 1988 ms. Trace:
I20260501 13:57:42.734072 1484 raft_consensus.cc:2468] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:42.804158 2409 raft_consensus.cc:2749] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:42.744254 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 9d38d75fcd494fc4a3420bcf15f2c70f (DEFAULT_TABLE table=default.Table_96180f484b044c0b85bb2f5d8a14d194 [id=48266956ca8b4274aea505435692daaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.805047 2093 ts_tablet_manager.cc:1929] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:42.748663 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9 (DEFAULT_TABLE table=default.Table_22d1c2cee41e4e9dad9e52bde00f1524 [id=a978daa811864093b075583f4e6bc4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.805374 2093 log.cc:1199] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/9a0d04810f2d465ebe2964797ea30538
I20260501 13:57:42.805938 2093 ts_tablet_manager.cc:1950] T 9a0d04810f2d465ebe2964797ea30538 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:42.806116 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d38d75fcd494fc4a3420bcf15f2c70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.806553 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9 (DEFAULT_TABLE table=default.Table_22d1c2cee41e4e9dad9e52bde00f1524 [id=a978daa811864093b075583f4e6bc4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.808655 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.826761 2028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:57:42.828795 2028 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:57:42.671530 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 6ea701a51a9440f283a6934ef5771979 (DEFAULT_TABLE table=default.Table_e3ff5a5e561a442d9af7f6c8317bd75b [id=a820b5b40cc04093a36c6602d01fd5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.831528 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.832270 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 240) took 1393 ms. Trace:
I20260501 13:57:42.832583 2093 rpcz_store.cc:276] 0501 13:57:41.438991 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:41.439397 (+ 406us) service_pool.cc:224] Handling call
0501 13:57:42.609823 (+1170426us) ts_tablet_manager.cc:1151] Deleting tablet 9a0d04810f2d465ebe2964797ea30538
0501 13:57:42.609868 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.763578 (+153710us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.804936 (+ 41358us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.832213 (+ 27277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24960,"tablet-delete.queue_time_us":1169867}
I20260501 13:57:42.833583 2093 tablet_replica.cc:333] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:42.804442 1899 rpcz_store.cc:276] 0501 13:57:40.814438 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.814574 (+ 136us) service_pool.cc:224] Handling call
0501 13:57:42.696812 (+1882238us) ts_tablet_manager.cc:1151] Deleting tablet 0c736959df334aee857dbaae0f7a1e1a
0501 13:57:42.696851 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.729936 (+ 33085us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.740722 (+ 10786us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:42.802438 (+ 61716us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1881717}
I20260501 13:57:42.835793 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 215) took 1956 ms. Trace:
I20260501 13:57:42.835863 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 0c736959df334aee857dbaae0f7a1e1a (table default.Table_56892e9ecd4b4b0b9133bc936b37c1c4 [id=09422f53fb424bdbaa00282b1775947d]) successfully deleted
I20260501 13:57:42.836088 1899 rpcz_store.cc:276] 0501 13:57:40.879072 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.879633 (+ 561us) service_pool.cc:224] Handling call
0501 13:57:42.835459 (+1955826us) ts_tablet_manager.cc:1151] Deleting tablet 1bb7fdcad3a54317b43bfd58f08e89d0
0501 13:57:42.835501 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.835741 (+ 240us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1947227}
I20260501 13:57:42.837137 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 216) took 1951 ms. Trace:
W20260501 13:57:42.837163 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 1bb7fdcad3a54317b43bfd58f08e89d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bb7fdcad3a54317b43bfd58f08e89d0 already in progress: creating tablet
I20260501 13:57:42.837416 1899 rpcz_store.cc:276] 0501 13:57:40.886042 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:40.886194 (+ 152us) service_pool.cc:224] Handling call
0501 13:57:42.836854 (+1950660us) ts_tablet_manager.cc:1151] Deleting tablet 1ab056da506f454984a07026341c534b
0501 13:57:42.836889 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.837093 (+ 204us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1948391}
W20260501 13:57:42.838438 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 1ab056da506f454984a07026341c534b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ab056da506f454984a07026341c534b already in progress: creating tablet
I20260501 13:57:42.839226 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea701a51a9440f283a6934ef5771979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.844404 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 37f57009c05f479481b59e49062608d5 (DEFAULT_TABLE table=default.Table_bd8182bfb41b4025b826d479090f8f09 [id=2b1b103ff1ad4207b67ce49e8c6802b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.845355 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f57009c05f479481b59e49062608d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.845255 1460 tablet_service.cc:1511] Processing CreateTablet for tablet ab08e04739cf42f3b7ae2e22eb5fbc36 (DEFAULT_TABLE table=default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.846161 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab08e04739cf42f3b7ae2e22eb5fbc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.846657 1760 tablet_service.cc:1511] Processing CreateTablet for tablet ab08e04739cf42f3b7ae2e22eb5fbc36 (DEFAULT_TABLE table=default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.847577 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab08e04739cf42f3b7ae2e22eb5fbc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.851876 1458 tablet_service.cc:1511] Processing CreateTablet for tablet 803c57a86b0242d0b536be679185245b (DEFAULT_TABLE table=default.Table_24c07f00f29c4f20835097d84de6e17a [id=5edf1e9aae2a4266bd94051ed9626558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.852828 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803c57a86b0242d0b536be679185245b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.853143 2028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:57:42.834254 2093 raft_consensus.cc:2243] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:42.866117 2409 consensus_queue.cc:1048] T 59508ee6bc6b47959d09f8f5aab89804 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:42.871418 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 9a0d04810f2d465ebe2964797ea30538 (table default.Table_a74b32dea54d43a2a8a1b6f2210f38d8 [id=eaa4c83258f94954af2e04eee604f6f0]) successfully deleted
I20260501 13:57:42.872309 2093 raft_consensus.cc:2272] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:42.838393 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 233) took 1292 ms. Trace:
I20260501 13:57:42.874766 1899 rpcz_store.cc:276] 0501 13:57:41.546329 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:41.546682 (+ 353us) service_pool.cc:224] Handling call
0501 13:57:42.838124 (+1291442us) ts_tablet_manager.cc:1151] Deleting tablet 8dbb619863dd405ab20ead6bbebf4d69
0501 13:57:42.838158 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.838351 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290930}
I20260501 13:57:42.875882 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 234) took 1193 ms. Trace:
I20260501 13:57:42.876191 1899 rpcz_store.cc:276] 0501 13:57:41.682639 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:41.683115 (+ 476us) service_pool.cc:224] Handling call
0501 13:57:42.875520 (+1192405us) ts_tablet_manager.cc:1151] Deleting tablet daca20eeafb141f2bf8b4233e0c8ea3b
0501 13:57:42.875560 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:42.875827 (+ 267us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191882}
W20260501 13:57:42.876706 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 8dbb619863dd405ab20ead6bbebf4d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dbb619863dd405ab20ead6bbebf4d69 already in progress: creating tablet
W20260501 13:57:42.877406 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet daca20eeafb141f2bf8b4233e0c8ea3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daca20eeafb141f2bf8b4233e0c8ea3b already in progress: creating tablet
W20260501 13:57:42.835196 1564 consensus_peers.cc:597] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260501 13:57:42.878571 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 43482.7us
I20260501 13:57:42.884940 1754 tablet_service.cc:1511] Processing CreateTablet for tablet 803c57a86b0242d0b536be679185245b (DEFAULT_TABLE table=default.Table_24c07f00f29c4f20835097d84de6e17a [id=5edf1e9aae2a4266bd94051ed9626558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.886032 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803c57a86b0242d0b536be679185245b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.885022 1882 tablet_bootstrap.cc:654] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:42.889865 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet f52ea8fb8ae0427d804001aed42cd67c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f52ea8fb8ae0427d804001aed42cd67c already in progress: creating tablet
I20260501 13:57:42.879076 2093 ts_tablet_manager.cc:1916] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:42.891916 1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 131397us
W20260501 13:57:42.893440 1717 leader_election.cc:343] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:42.895440 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 15a1f80cf7d24974b00c6cca7ae5783e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15a1f80cf7d24974b00c6cca7ae5783e already in progress: creating tablet
I20260501 13:57:42.898005 1717 leader_election.cc:304] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:42.913970 1882 tablet_bootstrap.cc:492] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:42.914388 1882 ts_tablet_manager.cc:1403] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.146s user 0.014s sys 0.000s
W20260501 13:57:42.921438 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53d9a8503c8d4bebb89ea493b99bf6fd consensus state has no leader
I20260501 13:57:42.922286 1132 master_service.cc:954] ReplaceTablet: received request to replace tablet f28db04617a449af8d87a554fb63e1e4 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:42.923369 2456 raft_consensus.cc:2749] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:42.926167 1603 tablet_service.cc:1511] Processing CreateTablet for tablet 37f57009c05f479481b59e49062608d5 (DEFAULT_TABLE table=default.Table_bd8182bfb41b4025b826d479090f8f09 [id=2b1b103ff1ad4207b67ce49e8c6802b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.926434 1601 tablet_service.cc:1511] Processing CreateTablet for tablet ab08e04739cf42f3b7ae2e22eb5fbc36 (DEFAULT_TABLE table=default.Table_f021eee779424a1d914441c79cc1e765 [id=a9e4d906858945bbb1814b12e4c346ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.926638 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 803c57a86b0242d0b536be679185245b (DEFAULT_TABLE table=default.Table_24c07f00f29c4f20835097d84de6e17a [id=5edf1e9aae2a4266bd94051ed9626558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:42.927244 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f57009c05f479481b59e49062608d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.927922 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803c57a86b0242d0b536be679185245b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.929469 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab08e04739cf42f3b7ae2e22eb5fbc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:42.930375 1883 tablet_bootstrap.cc:492] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:42.930824 1883 ts_tablet_manager.cc:1403] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.169s user 0.012s sys 0.000s
I20260501 13:57:42.948508 1881 tablet_bootstrap.cc:492] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:42.948930 1881 ts_tablet_manager.cc:1403] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.392s user 0.004s sys 0.005s
I20260501 13:57:42.896270 1899 tablet_replica.cc:333] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:42.949909 1899 raft_consensus.cc:2243] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:42.950480 1899 raft_consensus.cc:2272] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:42.982332 1096 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 45668.9us
I20260501 13:57:42.984575 1899 ts_tablet_manager.cc:1916] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:42.994771 1883 raft_consensus.cc:359] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.015491 1883 raft_consensus.cc:385] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:43.015874 1883 raft_consensus.cc:740] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:43.016654 1883 consensus_queue.cc:260] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:43.020373 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e2953dd57af41ea98d87d299e1a5f12 consensus state has no leader
I20260501 13:57:43.028820 1883 ts_tablet_manager.cc:1434] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.098s user 0.006s sys 0.000s
I20260501 13:57:43.032269 1883 tablet_bootstrap.cc:492] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:43.034384 1881 raft_consensus.cc:359] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.035357 1881 raft_consensus.cc:385] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:43.035669 1881 raft_consensus.cc:740] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:43.036443 1881 consensus_queue.cc:260] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:43.037109 1117 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 53d9a8503c8d4bebb89ea493b99bf6fd (table default.Table_33be6d10669d46efad19a4393e18553d [id=383d0757f1654b0b8de592bc1e1c3071], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 53d9a8503c8d4bebb89ea493b99bf6fd consensus state has no leader
I20260501 13:57:43.039790 1882 raft_consensus.cc:359] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.040411 1882 raft_consensus.cc:385] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:43.040650 1882 raft_consensus.cc:740] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:43.041293 1882 consensus_queue.cc:260] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.047886 1131 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ce141ed4961d49a8a07433b1dded9ce5" } modify_external_catalogs: true
I20260501 13:57:43.048389 1131 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ce141ed4961d49a8a07433b1dded9ce5" } modify_external_catalogs: true
I20260501 13:57:43.060079 1128 master_service.cc:954] ReplaceTablet: received request to replace tablet 15a1f80cf7d24974b00c6cca7ae5783e from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:43.067574 2409 raft_consensus.cc:493] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:43.068027 1130 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (table default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf]). Expected version 1 got 0
I20260501 13:57:43.068027 2409 raft_consensus.cc:515] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.068814 1130 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 6650472ec4a449c48c7957a9bbec88e3 (table default.Table_0d92e7b95c884bd5970e291f4189b0d9 [id=774a7eb90ba449b6aef225d819678439]). Expected version 1 got 0
I20260501 13:57:43.069598 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 3223417aa5594ef7942c6d90bdc8711c
I20260501 13:57:43.070575 2409 leader_election.cc:290] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:43.070554 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0669a229d6d54eacac9889ad18e06f30" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:43.070896 1126 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41700 (request call id 325) took 1321 ms. Trace:
W20260501 13:57:43.072656 1563 leader_election.cc:343] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 0669a229d6d54eacac9889ad18e06f30
W20260501 13:57:43.073504 1564 leader_election.cc:343] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:43.073843 1564 leader_election.cc:304] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:43.071192 1126 rpcz_store.cc:276] 0501 13:57:41.749380 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:41.761375 (+ 11995us) service_pool.cc:224] Handling call
0501 13:57:41.774389 (+ 13014us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663025470 00007f56630c431a 00007f5664dd846c 00007f5664ed3c46 00007f5662ae5024 00007f5662ae4fb2 00007f5662ae4ee1 00007f5662ae4e5c 00007f5662ae4112 00007f5660daebcc 00007f5660dae006
0501 13:57:41.778979 (+ 4590us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:57:41.779553 (+ 574us) catalog_manager.cc:2815] Modifying in-memory table state
0501 13:57:41.779665 (+ 112us) catalog_manager.cc:2824] Locking tablets
0501 13:57:42.255819 (+476154us) catalog_manager.cc:2838] Removing table and tablets from system table
0501 13:57:42.920690 (+664871us) catalog_manager.cc:2856] Removing table from by-name map
0501 13:57:42.920879 (+ 189us) catalog_manager.cc:2872] Committing in-memory state
0501 13:57:43.069515 (+148636us) catalog_manager.cc:2876] Aborting table tasks
0501 13:57:43.070830 (+ 1315us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:42.452573 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:42.452915 (+ 342us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:42.452937 (+ 22us) write_op.cc:435] Acquired schema lock
0501 13:57:42.452950 (+ 13us) tablet.cc:662] Decoding operations
0501 13:57:42.453275 (+ 325us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:42.453314 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:42.453336 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:57:42.453556 (+ 220us) tablet.cc:701] Row locks acquired
0501 13:57:42.453568 (+ 12us) write_op.cc:260] PREPARE: finished
0501 13:57:42.453672 (+ 104us) write_op.cc:270] Start()
0501 13:57:42.453746 (+ 74us) write_op.cc:276] Timestamp: P: 1777643862453696 usec, L: 0
0501 13:57:42.453762 (+ 16us) op_driver.cc:348] REPLICATION: starting
0501 13:57:42.454239 (+ 477us) log.cc:844] Serialized 728 byte log entry
0501 13:57:42.916228 (+461989us) op_driver.cc:464] REPLICATION: finished
0501 13:57:42.917824 (+ 1596us) write_op.cc:301] APPLY: starting
0501 13:57:42.917884 (+ 60us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:42.917993 (+ 109us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:42.918071 (+ 78us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:42.918288 (+ 217us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:42.918432 (+ 144us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0501 13:57:42.918494 (+ 62us) write_op.cc:312] APPLY: finished
0501 13:57:42.918667 (+ 173us) log.cc:844] Serialized 29 byte log entry
0501 13:57:42.918803 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:42.918866 (+ 63us) write_op.cc:454] Released schema lock
0501 13:57:42.918898 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:42.919135 (+ 237us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:42.919158 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":26990976,"child_traces":[["op",{"apply.queue_time_us":1442,"apply.run_cpu_time_us":1282,"apply.run_wall_time_us":1282,"num_ops":2,"prepare.queue_time_us":195138,"prepare.run_cpu_time_us":1856,"prepare.run_wall_time_us":1856,"replication_time_us":462352}]]}
I20260501 13:57:43.074438 2302 raft_consensus.cc:2749] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:43.076078 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet 3223417aa5594ef7942c6d90bdc8711c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:43.077644 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 3223417aa5594ef7942c6d90bdc8711c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3223417aa5594ef7942c6d90bdc8711c already in progress: creating tablet
I20260501 13:57:43.078116 1882 ts_tablet_manager.cc:1434] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.163s user 0.005s sys 0.000s
I20260501 13:57:43.078636 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet 3223417aa5594ef7942c6d90bdc8711c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:43.084570 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet 90a57bc58e854e469897a04b2d434d06 with delete_type TABLET_DATA_DELETED (replaced by tablet 3223417aa5594ef7942c6d90bdc8711c) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:43.085296 2047 tablet_replica.cc:333] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:43.085644 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a0d04810f2d465ebe2964797ea30538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:43.085963 2047 raft_consensus.cc:2243] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:43.086390 2047 raft_consensus.cc:2272] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:43.086697 1130 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.129:53385 (request call id 66) took 763 ms (client timeout 1000 ms). Trace:
I20260501 13:57:43.038412 1881 ts_tablet_manager.cc:1434] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.072s user 0.004s sys 0.000s
W20260501 13:57:43.087021 1130 rpcz_store.cc:269] 0501 13:57:42.323257 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:42.323406 (+ 149us) service_pool.cc:224] Handling call
0501 13:57:42.369799 (+ 46393us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663025470 00007f56630c431a 00007f5664dd846c 00007f5664ed23ee 00007f5662ad94a4 00007f5662ad9432 00007f5662ad9361 00007f5662ad92dc 00007f5662ad8592 00007f5660daebcc 00007f5660dae006
0501 13:57:43.086632 (+716833us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":103057152}
I20260501 13:57:43.088474 2047 ts_tablet_manager.cc:1916] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:43.089126 1881 tablet_bootstrap.cc:492] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:43.090896 1458 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ba63c153ef04db891cae46a71439455 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c1e0c30dce488baed9e046d90f0ba4) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:43.096468 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet 3223417aa5594ef7942c6d90bdc8711c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:43.100230 2302 consensus_queue.cc:1048] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:43.113260 1881 tablet_bootstrap.cc:654] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:43.125415 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_cbe7b051f36648d395177efd3e101e2d" } new_table_name: "default.Table_1ec94dd4afbb40d19a6f68aaefad8790" modify_external_catalogs: true
I20260501 13:57:43.129889 2093 ts_tablet_manager.cc:1929] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:43.130409 2093 log.cc:1199] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/863a10cfd14c4694b7c6141c0851167a
I20260501 13:57:43.132076 2093 ts_tablet_manager.cc:1950] T 863a10cfd14c4694b7c6141c0851167a P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:43.139266 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 238) took 1705 ms. Trace:
I20260501 13:57:43.145375 2093 rpcz_store.cc:276] 0501 13:57:41.433593 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:41.438721 (+ 5128us) service_pool.cc:224] Handling call
0501 13:57:42.833451 (+1394730us) ts_tablet_manager.cc:1151] Deleting tablet 863a10cfd14c4694b7c6141c0851167a
0501 13:57:42.833488 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:43.044724 (+211236us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:43.129734 (+ 85010us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:43.139206 (+ 9472us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1108,"tablet-delete.queue_time_us":1392955}
I20260501 13:57:43.154949 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 863a10cfd14c4694b7c6141c0851167a (table default.Table_95c99739e2574437b16501060ce816af [id=a6953bc50bab4e7f92adfee0ef034465]) successfully deleted
I20260501 13:57:43.156018 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 241) took 1609 ms. Trace:
I20260501 13:57:43.156330 2093 rpcz_store.cc:276] 0501 13:57:41.545653 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:41.545839 (+ 186us) service_pool.cc:224] Handling call
0501 13:57:43.155020 (+1609181us) ts_tablet_manager.cc:1151] Deleting tablet 8dbb619863dd405ab20ead6bbebf4d69
0501 13:57:43.155062 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:43.155294 (+ 232us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1608667}
W20260501 13:57:43.157362 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 8dbb619863dd405ab20ead6bbebf4d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dbb619863dd405ab20ead6bbebf4d69 already in progress: creating tablet
I20260501 13:57:43.158902 1881 tablet_bootstrap.cc:492] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:43.168350 1132 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet f28db04617a449af8d87a554fb63e1e4
I20260501 13:57:43.169693 1132 catalog_manager.cc:6741] ReplaceTablet: tablet f28db04617a449af8d87a554fb63e1e4 deleted and replaced by tablet 74af56cca54347bab67895e10ee17230
I20260501 13:57:43.174176 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet f28db04617a449af8d87a554fb63e1e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 74af56cca54347bab67895e10ee17230) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:43.179039 1881 ts_tablet_manager.cc:1403] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.001s
I20260501 13:57:43.181433 1881 raft_consensus.cc:359] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.185854 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 242) took 1503 ms. Trace:
I20260501 13:57:43.196010 2093 rpcz_store.cc:276] 0501 13:57:41.682109 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:41.683772 (+ 1663us) service_pool.cc:224] Handling call
0501 13:57:43.182327 (+1498555us) ts_tablet_manager.cc:1151] Deleting tablet daca20eeafb141f2bf8b4233e0c8ea3b
0501 13:57:43.182368 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:43.185789 (+ 3421us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1497949}
W20260501 13:57:43.197408 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet daca20eeafb141f2bf8b4233e0c8ea3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daca20eeafb141f2bf8b4233e0c8ea3b already in progress: creating tablet
I20260501 13:57:43.198428 1881 raft_consensus.cc:385] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:43.201395 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet f28db04617a449af8d87a554fb63e1e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 74af56cca54347bab67895e10ee17230) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:43.185776 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet f28db04617a449af8d87a554fb63e1e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 74af56cca54347bab67895e10ee17230) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:43.202601 2093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 244) took 1200 ms. Trace:
I20260501 13:57:43.202976 1881 raft_consensus.cc:740] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:43.206754 2093 rpcz_store.cc:276] 0501 13:57:42.002446 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:42.003009 (+ 563us) service_pool.cc:224] Handling call
0501 13:57:43.202224 (+1199215us) ts_tablet_manager.cc:1151] Deleting tablet f52ea8fb8ae0427d804001aed42cd67c
0501 13:57:43.202266 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:43.202537 (+ 271us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1198420}
I20260501 13:57:43.203717 1881 consensus_queue.cc:260] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:43.210047 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet f52ea8fb8ae0427d804001aed42cd67c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f52ea8fb8ae0427d804001aed42cd67c already in progress: creating tablet
W20260501 13:57:43.212365 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 15a1f80cf7d24974b00c6cca7ae5783e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15a1f80cf7d24974b00c6cca7ae5783e already in progress: creating tablet
W20260501 13:57:43.223269 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 38c3eb388d474d718554775d65f58d78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38c3eb388d474d718554775d65f58d78 already in progress: creating tablet
I20260501 13:57:43.215222 1881 ts_tablet_manager.cc:1434] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.036s user 0.001s sys 0.005s
I20260501 13:57:43.224498 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ecd622b82fa9463198fe36afb1458343" } new_table_name: "default.Table_51001cce4a264aacb220489c7c4bffd0" modify_external_catalogs: true
W20260501 13:57:43.225040 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 3223417aa5594ef7942c6d90bdc8711c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3223417aa5594ef7942c6d90bdc8711c already in progress: creating tablet
W20260501 13:57:43.232870 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet f28db04617a449af8d87a554fb63e1e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f28db04617a449af8d87a554fb63e1e4 already in progress: creating tablet
I20260501 13:57:43.227413 1881 tablet_bootstrap.cc:492] T 9018d70c734a429aa59565ad67a46acc P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:43.255889 1881 tablet_bootstrap.cc:654] T 9018d70c734a429aa59565ad67a46acc P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:43.266113 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_b12841aa2a8240a0b6e3523a0e583b98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:43.079496 1882 tablet_bootstrap.cc:492] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
W20260501 13:57:43.267753 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b12841aa2a8240a0b6e3523a0e583b98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:43.277540 1883 tablet_bootstrap.cc:654] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:43.279090 1125 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_a1116dca3a314f4b8508c62da88fbdc9" } modify_external_catalogs: true
I20260501 13:57:43.279615 1125 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_a1116dca3a314f4b8508c62da88fbdc9" } modify_external_catalogs: true
I20260501 13:57:43.296345 1881 tablet_bootstrap.cc:492] T 9018d70c734a429aa59565ad67a46acc P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:43.296991 1881 ts_tablet_manager.cc:1403] T 9018d70c734a429aa59565ad67a46acc P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.070s user 0.008s sys 0.006s
I20260501 13:57:43.299386 1881 raft_consensus.cc:359] T 9018d70c734a429aa59565ad67a46acc P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.303084 1881 raft_consensus.cc:385] T 9018d70c734a429aa59565ad67a46acc P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:43.303406 1881 raft_consensus.cc:740] T 9018d70c734a429aa59565ad67a46acc P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:43.312026 1881 consensus_queue.cc:260] T 9018d70c734a429aa59565ad67a46acc P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.316504 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_6ed6c8b736e9485789f83a5b49e1d994"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:43.318421 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ed6c8b736e9485789f83a5b49e1d994 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:43.331210 1881 ts_tablet_manager.cc:1434] T 9018d70c734a429aa59565ad67a46acc P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20260501 13:57:43.333374 1881 tablet_bootstrap.cc:492] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:43.367136 1480 raft_consensus.cc:3060] T 9018d70c734a429aa59565ad67a46acc P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:43.396497 2047 ts_tablet_manager.cc:1929] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:43.400748 2047 log.cc:1199] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/90a57bc58e854e469897a04b2d434d06
I20260501 13:57:43.401985 2047 ts_tablet_manager.cc:1950] T 90a57bc58e854e469897a04b2d434d06 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:43.403697 1881 tablet_bootstrap.cc:654] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:43.404289 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 90a57bc58e854e469897a04b2d434d06 (table default.Table_f8e9d88347b54e2da903776319f1c181 [id=b3e0852bf1bc4eedad8b69f7e4cf0785]) successfully deleted
W20260501 13:57:43.406646 1117 catalog_manager.cc:4969] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 9a0d04810f2d465ebe2964797ea30538 because the tablet was not found. No further retry: Not found: Tablet not found: 9a0d04810f2d465ebe2964797ea30538
I20260501 13:57:43.408051 2047 tablet_replica.cc:333] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:43.411417 2047 raft_consensus.cc:2243] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:43.415782 2047 raft_consensus.cc:2272] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:43.420226 2047 ts_tablet_manager.cc:1916] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:43.417338 1881 tablet_bootstrap.cc:492] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:43.424592 1881 ts_tablet_manager.cc:1403] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.091s user 0.012s sys 0.002s
W20260501 13:57:43.389750 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 131510us
I20260501 13:57:43.427165 1881 raft_consensus.cc:359] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.427839 1881 raft_consensus.cc:385] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:43.428088 1881 raft_consensus.cc:740] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:43.430742 1881 consensus_queue.cc:260] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.433099 1881 ts_tablet_manager.cc:1434] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260501 13:57:43.434011 1881 tablet_bootstrap.cc:492] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:43.441975 2423 raft_consensus.cc:493] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:43.442518 2423 raft_consensus.cc:515] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.361112 2496 tablet.cc:1724] T c33a1f423bdb44f5843daffe8f73e74a P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:43.446779 2423 leader_election.cc:290] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:43.446766 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe89d656956440e8d0c1de0b01ce486" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:43.449877 1881 tablet_bootstrap.cc:654] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:43.292320 2456 raft_consensus.cc:493] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:43.454120 2456 raft_consensus.cc:515] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.456140 2456 leader_election.cc:290] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:43.446244 1780 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe89d656956440e8d0c1de0b01ce486" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:43.458534 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cd353ca044465a6692656412e7d82" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:43.458360 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cd353ca044465a6692656412e7d82" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:43.459870 1717 leader_election.cc:343] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:43.459829 1418 leader_election.cc:304] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
W20260501 13:57:43.461769 1719 leader_election.cc:343] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:43.462143 1719 leader_election.cc:304] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:43.463728 2456 raft_consensus.cc:2749] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:43.464450 2423 raft_consensus.cc:2749] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:43.465503 1881 tablet_bootstrap.cc:492] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:43.465894 1881 ts_tablet_manager.cc:1403] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.008s
I20260501 13:57:43.467530 2302 raft_consensus.cc:493] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:43.468047 2302 raft_consensus.cc:515] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.470505 2302 leader_election.cc:290] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:43.474350 1881 raft_consensus.cc:359] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.475068 1881 raft_consensus.cc:385] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:43.475359 1881 raft_consensus.cc:740] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:43.476087 1881 consensus_queue.cc:260] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.478590 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0356b19f0aa4197a41cc66d4440bffb" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:43.480089 1881 ts_tablet_manager.cc:1434] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260501 13:57:43.480886 1881 tablet_bootstrap.cc:492] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:43.353840 2503 tablet.cc:1724] T c33a1f423bdb44f5843daffe8f73e74a P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 13:57:43.480664 1564 leader_election.cc:343] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:43.505633 2047 ts_tablet_manager.cc:1929] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260501 13:57:43.507601 1563 leader_election.cc:343] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: e0356b19f0aa4197a41cc66d4440bffb
I20260501 13:57:43.508003 1563 leader_election.cc:304] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:43.509308 2499 raft_consensus.cc:2749] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:43.510854 1882 tablet_bootstrap.cc:654] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:43.518656 2047 log.cc:1199] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/1ba63c153ef04db891cae46a71439455
I20260501 13:57:43.519469 2047 ts_tablet_manager.cc:1950] T 1ba63c153ef04db891cae46a71439455 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:43.520622 2496 tablet.cc:1724] T 6650472ec4a449c48c7957a9bbec88e3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:43.521224 1899 ts_tablet_manager.cc:1929] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:43.521593 1899 log.cc:1199] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/38c3eb388d474d718554775d65f58d78
I20260501 13:57:43.522186 1899 ts_tablet_manager.cc:1950] T 38c3eb388d474d718554775d65f58d78 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:43.526865 2456 raft_consensus.cc:493] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:43.527283 2456 raft_consensus.cc:515] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.529256 2456 leader_election.cc:290] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:43.530836 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56cee07054a64e199a0bfc26b63eca2a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:43.532348 1719 leader_election.cc:343] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:43.541606 1883 tablet_bootstrap.cc:492] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:43.542117 1883 ts_tablet_manager.cc:1403] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.510s user 0.006s sys 0.005s
I20260501 13:57:43.548051 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56cee07054a64e199a0bfc26b63eca2a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:43.549341 1717 leader_election.cc:343] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:43.549728 1717 leader_election.cc:304] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:43.498219 2521 tablet.cc:1724] T c33a1f423bdb44f5843daffe8f73e74a P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:43.556457 1882 tablet_bootstrap.cc:492] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:43.556849 1882 ts_tablet_manager.cc:1403] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.478s user 0.009s sys 0.000s
W20260501 13:57:43.558429 1117 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.189:45897, user_credentials={real_user=slave}} blocked reactor thread for 68638.1us
I20260501 13:57:43.558840 2456 raft_consensus.cc:2749] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:43.559283 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 1ba63c153ef04db891cae46a71439455 (table default.Table_66a4e496efa34deaa6c793a210a510ff [id=e8b19350d702499a82f41a6cce14dbd5]) successfully deleted
W20260501 13:57:43.559888 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet f28db04617a449af8d87a554fb63e1e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f28db04617a449af8d87a554fb63e1e4 already in progress: creating tablet
I20260501 13:57:43.561367 2499 consensus_queue.cc:1048] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:43.565464 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 967658b1be204c638fc34983293ef5ab consensus state has no leader
I20260501 13:57:43.567294 1899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 246) took 1117 ms. Trace:
I20260501 13:57:43.567622 1899 rpcz_store.cc:276] 0501 13:57:42.450388 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:42.450540 (+ 152us) service_pool.cc:224] Handling call
0501 13:57:42.896125 (+445585us) ts_tablet_manager.cc:1151] Deleting tablet 38c3eb388d474d718554775d65f58d78
0501 13:57:42.896161 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:43.270936 (+374775us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:43.521103 (+250167us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:43.567232 (+ 46129us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":444960}
I20260501 13:57:43.569998 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 38c3eb388d474d718554775d65f58d78 (table default.Table_392605e427d14287b4fccdd9b198c6a2 [id=d6ca68cf0d954555b422b90e6502d6cc]) successfully deleted
W20260501 13:57:43.571134 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 3223417aa5594ef7942c6d90bdc8711c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3223417aa5594ef7942c6d90bdc8711c already in progress: creating tablet
W20260501 13:57:43.571760 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet f28db04617a449af8d87a554fb63e1e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f28db04617a449af8d87a554fb63e1e4 already in progress: creating tablet
I20260501 13:57:43.581413 2520 raft_consensus.cc:493] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:43.582063 2520 raft_consensus.cc:515] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.585479 2520 leader_election.cc:290] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:43.594765 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33a355897309420c97b1257afaefe8d1" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:43.596171 1564 leader_election.cc:343] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:43.602903 1881 tablet_bootstrap.cc:654] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:43.603091 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81daa397384d4afb816071ee665a1dcd consensus state has no leader
I20260501 13:57:43.604358 2528 tablet.cc:1724] T 9018d70c734a429aa59565ad67a46acc P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:43.630987 1124 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 7d399de014d74cab8df14f44787ca2f7 (table default.Table_aa47824eda84455d9d52df61514a253e [id=81117c514ffd4ab6a8d4dde20d7559e4]). Expected version 1 got 0
I20260501 13:57:43.631821 1124 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715]). Expected version 1 got 0
I20260501 13:57:43.633077 1606 tablet_service.cc:1511] Processing CreateTablet for tablet f0a81103ac464d9680ed1ddb17734060 (DEFAULT_TABLE table=default.Table_8c3198f65f4b4ef78045ab574046fc33 [id=ccdd812dcf014fecad92562e373530bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.634083 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a81103ac464d9680ed1ddb17734060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.635079 1456 tablet_service.cc:1511] Processing CreateTablet for tablet f0a81103ac464d9680ed1ddb17734060 (DEFAULT_TABLE table=default.Table_8c3198f65f4b4ef78045ab574046fc33 [id=ccdd812dcf014fecad92562e373530bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.635995 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a81103ac464d9680ed1ddb17734060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.640750 1601 tablet_service.cc:1511] Processing CreateTablet for tablet c35118ffca2d471b9d464cb11cbe4d75 (DEFAULT_TABLE table=default.Table_628b3fff3b534cdf8d44d255ed5f241f [id=2119836f83654c2c8d429049089a1397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.641783 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35118ffca2d471b9d464cb11cbe4d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.643055 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 0ff55996178045c0981184330b903bb8 (DEFAULT_TABLE table=default.Table_e2f41fc3135c458e9fda2d60189b4a23 [id=d286fd8e6cb040eaa6f01a571c3e7d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.643944 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff55996178045c0981184330b903bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.646730 1603 tablet_service.cc:1511] Processing CreateTablet for tablet 7b53cee2e0904c86a8c5cca35ba81cb5 (DEFAULT_TABLE table=default.Table_f9cbd6f5e56b422c897fcbd0b6c7e718 [id=ed2dc400b0a3418698c613fd621a7deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.647619 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b53cee2e0904c86a8c5cca35ba81cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.648450 1458 tablet_service.cc:1511] Processing CreateTablet for tablet 0ff55996178045c0981184330b903bb8 (DEFAULT_TABLE table=default.Table_e2f41fc3135c458e9fda2d60189b4a23 [id=d286fd8e6cb040eaa6f01a571c3e7d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.648661 1455 tablet_service.cc:1511] Processing CreateTablet for tablet c35118ffca2d471b9d464cb11cbe4d75 (DEFAULT_TABLE table=default.Table_628b3fff3b534cdf8d44d255ed5f241f [id=2119836f83654c2c8d429049089a1397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.650015 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 7b53cee2e0904c86a8c5cca35ba81cb5 (DEFAULT_TABLE table=default.Table_f9cbd6f5e56b422c897fcbd0b6c7e718 [id=ed2dc400b0a3418698c613fd621a7deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.651000 2521 tablet.cc:1724] T 6650472ec4a449c48c7957a9bbec88e3 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:43.632439 1124 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 6650472ec4a449c48c7957a9bbec88e3 (table default.Table_0d92e7b95c884bd5970e291f4189b0d9 [id=774a7eb90ba449b6aef225d819678439]). Expected version 1 got 0
I20260501 13:57:43.634019 1126 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 96e937b3a0b5451e8eaf4d5c47c17620 (table default.Table_b29786a52abd4fad9a2337d7c76fbbc4 [id=05998ead56ed454dab9a71ef2ac9cb58]). Expected version 1 got 0
I20260501 13:57:43.654665 1882 raft_consensus.cc:359] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.655354 1882 raft_consensus.cc:385] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:43.655709 1882 raft_consensus.cc:740] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:43.660296 1480 raft_consensus.cc:3060] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:43.662851 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 6399ac6a0d044ad2aed95ba30e6659b8 (DEFAULT_TABLE table=default.Table_7bbaf820afc746198b26a791c87cb736 [id=7bde73227e33452984fc7154a5547299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.665616 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 6399ac6a0d044ad2aed95ba30e6659b8 (DEFAULT_TABLE table=default.Table_7bbaf820afc746198b26a791c87cb736 [id=7bde73227e33452984fc7154a5547299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.666635 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6399ac6a0d044ad2aed95ba30e6659b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.666998 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff55996178045c0981184330b903bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.667578 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6399ac6a0d044ad2aed95ba30e6659b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.670717 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b53cee2e0904c86a8c5cca35ba81cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.672430 1778 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33a355897309420c97b1257afaefe8d1" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:43.673132 1778 raft_consensus.cc:2468] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:43.678074 1563 leader_election.cc:304] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:43.679563 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_3f7be42c55ff49c0ac1a846980e70e15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:43.681178 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f7be42c55ff49c0ac1a846980e70e15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:43.683789 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_b92aef0dd8d843679f848b994f328edf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:43.685392 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b92aef0dd8d843679f848b994f328edf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:43.704377 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35118ffca2d471b9d464cb11cbe4d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.705811 1465 tablet_service.cc:1511] Processing CreateTablet for tablet df1c0b6173654535bd01af7aa3dc6ce9 (DEFAULT_TABLE table=default.Table_31a3f0b22f09450faa99203169288584 [id=6566d5f1bfad40d29ec026d68a935cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.707105 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 71ac54444a414e25bcd05452edfbf5ba (DEFAULT_TABLE table=default.Table_60c762a708a54d729a3eb53a841dfe12 [id=2aa7cf25459145adb3f344a5823fba83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.708455 2423 raft_consensus.cc:493] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:43.708956 2423 raft_consensus.cc:515] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.709560 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1c0b6173654535bd01af7aa3dc6ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.708894 1460 tablet_service.cc:1511] Processing CreateTablet for tablet 4802dafc3e6b4874898b7dee694fc76e (DEFAULT_TABLE table=default.Table_fb6bb57c871e4855aba95f3b430a3fc7 [id=09adc59479f44d358ed186c100b690b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.715620 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4802dafc3e6b4874898b7dee694fc76e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.719735 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ac54444a414e25bcd05452edfbf5ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:43.715641 1421 leader_election.cc:343] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Not found: Tablet not found: b98187c7843e42b09d55c851117c4ac8
I20260501 13:57:43.732533 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 7d399de014d74cab8df14f44787ca2f7 (table default.Table_aa47824eda84455d9d52df61514a253e [id=81117c514ffd4ab6a8d4dde20d7559e4]). Expected version 1 got 0
I20260501 13:57:43.743428 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:43.744606 2047 tablet_replica.cc:333] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:43.745329 2047 raft_consensus.cc:2243] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:43.733510 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715]). Expected version 1 got 0
I20260501 13:57:43.745754 2047 raft_consensus.cc:2272] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:43.746177 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 56cee07054a64e199a0bfc26b63eca2a (table default.Table_9b824b59406d478884838d29a5d690f8 [id=389132d3db474d008cc0ee9d758b4fec]). Expected version 1 got 0
I20260501 13:57:43.748502 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 6650472ec4a449c48c7957a9bbec88e3 (table default.Table_0d92e7b95c884bd5970e291f4189b0d9 [id=774a7eb90ba449b6aef225d819678439]). Expected version 1 got 0
I20260501 13:57:43.768926 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_d22010815be1426a92087cb895458676" } new_table_name: "default.Table_be2275f0abfb43abb3bd0b1e128d7fef" modify_external_catalogs: true
I20260501 13:57:43.792845 1883 raft_consensus.cc:359] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.793694 1883 raft_consensus.cc:385] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:43.794013 1883 raft_consensus.cc:740] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:43.691148 1603 tablet_service.cc:1511] Processing CreateTablet for tablet df1c0b6173654535bd01af7aa3dc6ce9 (DEFAULT_TABLE table=default.Table_31a3f0b22f09450faa99203169288584 [id=6566d5f1bfad40d29ec026d68a935cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.686744 1882 consensus_queue.cc:260] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.699348 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 4802dafc3e6b4874898b7dee694fc76e (DEFAULT_TABLE table=default.Table_fb6bb57c871e4855aba95f3b430a3fc7 [id=09adc59479f44d358ed186c100b690b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.795823 2511 raft_consensus.cc:2804] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:43.796173 2511 raft_consensus.cc:493] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:43.796252 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1c0b6173654535bd01af7aa3dc6ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.796476 2511 raft_consensus.cc:3060] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:43.797097 1882 ts_tablet_manager.cc:1434] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.240s user 0.005s sys 0.001s
I20260501 13:57:43.703542 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 71ac54444a414e25bcd05452edfbf5ba (DEFAULT_TABLE table=default.Table_60c762a708a54d729a3eb53a841dfe12 [id=2aa7cf25459145adb3f344a5823fba83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.799314 1882 tablet_bootstrap.cc:492] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:43.803318 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ac54444a414e25bcd05452edfbf5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.806970 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4802dafc3e6b4874898b7dee694fc76e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.807210 1130 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_f9cbd6f5e56b422c897fcbd0b6c7e718" } new_table_name: "default.Table_83fc7d081dd24a7fa7e4e62073257a70" modify_external_catalogs: true
W20260501 13:57:43.728684 1564 consensus_peers.cc:597] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:43.729080 2520 consensus_queue.cc:1048] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:43.809049 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 80451.9us
I20260501 13:57:43.813748 1609 tablet_service.cc:1511] Processing CreateTablet for tablet d38d22f2f5b2469a892967b82edbaee5 (DEFAULT_TABLE table=default.Table_79fb3bc412df49ea8ed51d3a8a6f91f5 [id=8cefbb1691bf4ef5852870b1fef2e2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.814702 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38d22f2f5b2469a892967b82edbaee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.819506 1601 tablet_service.cc:1511] Processing CreateTablet for tablet a000b129a6ab429889b543c20750a493 (DEFAULT_TABLE table=default.Table_31b7fbe1002b47b0adb3e3abb1517807 [id=e2c1080f71804f5b8f68ac9fbcfeb7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.820425 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a000b129a6ab429889b543c20750a493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.819849 2511 raft_consensus.cc:515] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.824973 1882 tablet_bootstrap.cc:654] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:43.831360 1131 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet e8efee17e2c34caf98c0ede01d2a943a
I20260501 13:57:43.832655 1131 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41700 (request call id 340) took 1178 ms. Trace:
I20260501 13:57:43.832933 1131 rpcz_store.cc:276] 0501 13:57:42.654166 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:43.020983 (+366817us) service_pool.cc:224] Handling call
0501 13:57:43.047773 (+ 26790us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663025470 00007f56630c431a 00007f5664dd846c 00007f5664ed3c46 00007f5662ae5024 00007f5662ae4fb2 00007f5662ae4ee1 00007f5662ae4e5c 00007f5662ae4112 00007f5660daebcc 00007f5660dae006
0501 13:57:43.048765 (+ 992us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:57:43.049926 (+ 1161us) catalog_manager.cc:2815] Modifying in-memory table state
0501 13:57:43.050733 (+ 807us) catalog_manager.cc:2824] Locking tablets
0501 13:57:43.051488 (+ 755us) catalog_manager.cc:2838] Removing table and tablets from system table
0501 13:57:43.168066 (+116578us) catalog_manager.cc:2856] Removing table from by-name map
0501 13:57:43.168259 (+ 193us) catalog_manager.cc:2872] Committing in-memory state
0501 13:57:43.831270 (+663011us) catalog_manager.cc:2876] Aborting table tasks
0501 13:57:43.832600 (+ 1330us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:43.054668 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:43.055049 (+ 381us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:43.055073 (+ 24us) write_op.cc:435] Acquired schema lock
0501 13:57:43.055084 (+ 11us) tablet.cc:662] Decoding operations
0501 13:57:43.055372 (+ 288us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:43.055407 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:43.055430 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:57:43.055700 (+ 270us) tablet.cc:701] Row locks acquired
0501 13:57:43.055713 (+ 13us) write_op.cc:260] PREPARE: finished
0501 13:57:43.055815 (+ 102us) write_op.cc:270] Start()
0501 13:57:43.055906 (+ 91us) write_op.cc:276] Timestamp: P: 1777643863055784 usec, L: 0
0501 13:57:43.055921 (+ 15us) op_driver.cc:348] REPLICATION: starting
0501 13:57:43.056364 (+ 443us) log.cc:844] Serialized 759 byte log entry
0501 13:57:43.165688 (+109324us) op_driver.cc:464] REPLICATION: finished
0501 13:57:43.166021 (+ 333us) write_op.cc:301] APPLY: starting
0501 13:57:43.166070 (+ 49us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:43.166689 (+ 619us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:43.166779 (+ 90us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:43.167006 (+ 227us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:43.167131 (+ 125us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0501 13:57:43.167172 (+ 41us) write_op.cc:312] APPLY: finished
0501 13:57:43.167330 (+ 158us) log.cc:844] Serialized 29 byte log entry
0501 13:57:43.167451 (+ 121us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:43.167506 (+ 55us) write_op.cc:454] Released schema lock
0501 13:57:43.167537 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:43.167742 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:43.167765 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":57292032,"child_traces":[["op",{"apply.queue_time_us":225,"apply.run_cpu_time_us":1689,"apply.run_wall_time_us":1689,"num_ops":2,"prepare.queue_time_us":121,"prepare.run_cpu_time_us":1955,"prepare.run_wall_time_us":1954,"raft.queue_time_us":156,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":11,"replication_time_us":109705}]]}
I20260501 13:57:43.839514 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_1adc3e421970491abbba47fa8fe9ab5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:43.841172 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1adc3e421970491abbba47fa8fe9ab5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:43.847366 1462 tablet_service.cc:1511] Processing CreateTablet for tablet d38d22f2f5b2469a892967b82edbaee5 (DEFAULT_TABLE table=default.Table_79fb3bc412df49ea8ed51d3a8a6f91f5 [id=8cefbb1691bf4ef5852870b1fef2e2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.848263 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38d22f2f5b2469a892967b82edbaee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.849875 2047 ts_tablet_manager.cc:1916] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:43.864372 1127 catalog_manager.cc:5671] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:43.865873 1127 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 406e1af8b7d9488f88697206daa87c05 (table default.Table_759f3009542147de94c73784e1698246 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]). Expected version 1 got 0
I20260501 13:57:43.866792 1127 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 6650472ec4a449c48c7957a9bbec88e3 (table default.Table_0d92e7b95c884bd5970e291f4189b0d9 [id=774a7eb90ba449b6aef225d819678439]). Expected version 1 got 0
I20260501 13:57:43.867421 1127 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:43.868093 1127 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715]). Expected version 1 got 0
I20260501 13:57:43.869045 1127 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (table default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf]). Expected version 1 got 0
I20260501 13:57:43.869740 1127 catalog_manager.cc:5671] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:43.726730 2423 leader_election.cc:290] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:43.879812 1456 tablet_service.cc:1511] Processing CreateTablet for tablet a000b129a6ab429889b543c20750a493 (DEFAULT_TABLE table=default.Table_31b7fbe1002b47b0adb3e3abb1517807 [id=e2c1080f71804f5b8f68ac9fbcfeb7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.880404 2550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.188:38815 every 8 connection(s)
I20260501 13:57:43.881167 2028 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.188:38815
W20260501 13:57:43.881553 1418 leader_election.cc:343] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: b98187c7843e42b09d55c851117c4ac8
I20260501 13:57:43.881999 1418 leader_election.cc:304] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:43.885269 2028 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
I20260501 13:57:43.887434 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 2028
I20260501 13:57:43.794795 1883 consensus_queue.cc:260] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.891748 2536 raft_consensus.cc:493] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:43.892159 2536 raft_consensus.cc:515] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:43.894006 2536 leader_election.cc:290] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:43.895504 1127 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 96e937b3a0b5451e8eaf4d5c47c17620 (table default.Table_b29786a52abd4fad9a2337d7c76fbbc4 [id=05998ead56ed454dab9a71ef2ac9cb58]). Expected version 1 got 0
I20260501 13:57:43.902638 1465 tablet_service.cc:1511] Processing CreateTablet for tablet bdc412d7b5634ef2893b59467be4fa92 (DEFAULT_TABLE table=default.Table_9f3c5df0faf64dedabec83d1c7d3dd82 [id=606cf51052b443f7aacbd78885edd3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.904819 1125 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet eec4bce26de14ea1915faea94490d665
I20260501 13:57:43.906565 1125 master_service.cc:954] ReplaceTablet: received request to replace tablet 15a1f80cf7d24974b00c6cca7ae5783e from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:43.908233 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_00e5296f8b75468ba13600c8afa47ce7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:43.909698 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00e5296f8b75468ba13600c8afa47ce7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:43.914829 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet e8efee17e2c34caf98c0ede01d2a943a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:43 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:43.915864 1457 tablet_service.cc:1558] Processing DeleteTablet for tablet eec4bce26de14ea1915faea94490d665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:43 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:43.917452 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 71f7bd151fa64fe5a80e756a95a375c5 (DEFAULT_TABLE table=default.Table_fb33a28f96da45c29f6e68281ec90b8d [id=1beb747ddd3841f8a489e91ceb05c8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.936689 1883 ts_tablet_manager.cc:1434] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.394s user 0.006s sys 0.000s
I20260501 13:57:43.898197 2536 raft_consensus.cc:2749] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:43.949321 1754 tablet_service.cc:1511] Processing CreateTablet for tablet f0a81103ac464d9680ed1ddb17734060 (DEFAULT_TABLE table=default.Table_8c3198f65f4b4ef78045ab574046fc33 [id=ccdd812dcf014fecad92562e373530bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.950294 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a81103ac464d9680ed1ddb17734060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.950295 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a000b129a6ab429889b543c20750a493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.952102 1755 tablet_service.cc:1511] Processing CreateTablet for tablet 7b53cee2e0904c86a8c5cca35ba81cb5 (DEFAULT_TABLE table=default.Table_f9cbd6f5e56b422c897fcbd0b6c7e718 [id=ed2dc400b0a3418698c613fd621a7deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.952966 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b53cee2e0904c86a8c5cca35ba81cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.954033 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 6399ac6a0d044ad2aed95ba30e6659b8 (DEFAULT_TABLE table=default.Table_7bbaf820afc746198b26a791c87cb736 [id=7bde73227e33452984fc7154a5547299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.954910 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6399ac6a0d044ad2aed95ba30e6659b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.955961 1752 tablet_service.cc:1511] Processing CreateTablet for tablet c35118ffca2d471b9d464cb11cbe4d75 (DEFAULT_TABLE table=default.Table_628b3fff3b534cdf8d44d255ed5f241f [id=2119836f83654c2c8d429049089a1397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.956781 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35118ffca2d471b9d464cb11cbe4d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.958748 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdc412d7b5634ef2893b59467be4fa92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.959652 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71f7bd151fa64fe5a80e756a95a375c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:43.960708 1123 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_a7b6759170be4cce860b79126e7bbc29 [id=454925118d6943458e68bb0089610a15] alter complete (version 2)
W20260501 13:57:43.969157 1124 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.131:42895 (request call id 67) took 1684 ms (client timeout 1000 ms). Trace:
W20260501 13:57:43.969501 1124 rpcz_store.cc:269] 0501 13:57:42.284598 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:42.324604 (+ 40006us) service_pool.cc:224] Handling call
0501 13:57:42.374817 (+ 50213us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663025470 00007f56630c431a 00007f5664dd846c 00007f5664ed23ee 00007f5662ad94a4 00007f5662ad9432 00007f5662ad9361 00007f5662ad92dc 00007f5662ad8592 00007f5660daebcc 00007f5660dae006
0501 13:57:43.969095 (+1594278us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":94,"reported_tablets":10,"spinlock_wait_cycles":75046272}
W20260501 13:57:43.974807 1123 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.130:35289 (request call id 69) took 1638 ms (client timeout 1000 ms). Trace:
W20260501 13:57:43.976908 1130 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b53cee2e0904c86a8c5cca35ba81cb5 consensus state has no leader
W20260501 13:57:43.975121 1123 rpcz_store.cc:269] 0501 13:57:42.336229 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:42.336336 (+ 107us) service_pool.cc:224] Handling call
0501 13:57:42.365990 (+ 29654us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663025470 00007f56630c431a 00007f5664dd846c 00007f5664ed23ee 00007f5662ad94a4 00007f5662ad9432 00007f5662ad9361 00007f5662ad92dc 00007f5662ad8592 00007f5660daebcc 00007f5660dae006
0501 13:57:43.974747 (+1608757us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:42.500837 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:42.501187 (+ 350us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:42.501210 (+ 23us) write_op.cc:435] Acquired schema lock
0501 13:57:42.501222 (+ 12us) tablet.cc:662] Decoding operations
0501 13:57:42.501493 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:42.501527 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:42.501549 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:57:42.501782 (+ 233us) tablet.cc:701] Row locks acquired
0501 13:57:42.501793 (+ 11us) write_op.cc:260] PREPARE: finished
0501 13:57:42.501890 (+ 97us) write_op.cc:270] Start()
0501 13:57:42.511475 (+ 9585us) write_op.cc:276] Timestamp: P: 1777643862501920 usec, L: 0
0501 13:57:42.511501 (+ 26us) op_driver.cc:348] REPLICATION: starting
0501 13:57:42.512025 (+ 524us) log.cc:844] Serialized 907 byte log entry
0501 13:57:43.054712 (+542687us) op_driver.cc:464] REPLICATION: finished
0501 13:57:43.056471 (+ 1759us) write_op.cc:301] APPLY: starting
0501 13:57:43.056529 (+ 58us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:43.056627 (+ 98us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:43.056848 (+ 221us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:43.057097 (+ 249us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:43.057223 (+ 126us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0501 13:57:43.057263 (+ 40us) write_op.cc:312] APPLY: finished
0501 13:57:43.057418 (+ 155us) log.cc:844] Serialized 29 byte log entry
0501 13:57:43.057551 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:43.057609 (+ 58us) write_op.cc:454] Released schema lock
0501 13:57:43.057640 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:43.057837 (+ 197us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:43.057870 (+ 33us) write_op.cc:454] Released schema lock
Related trace 'op':
0501 13:57:43.065877 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:43.066249 (+ 372us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:43.066273 (+ 24us) write_op.cc:435] Acquired schema lock
0501 13:57:43.066287 (+ 14us) tablet.cc:662] Decoding operations
0501 13:57:43.066541 (+ 254us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:43.066576 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:43.066598 (+ 22us) tablet.cc:685] Acquiring locks for 1 operations
0501 13:57:43.066729 (+ 131us) tablet.cc:701] Row locks acquired
0501 13:57:43.066743 (+ 14us) write_op.cc:260] PREPARE: finished
0501 13:57:43.066846 (+ 103us) write_op.cc:270] Start()
0501 13:57:43.066931 (+ 85us) write_op.cc:276] Timestamp: P: 1777643863066815 usec, L: 0
0501 13:57:43.066947 (+ 16us) op_driver.cc:348] REPLICATION: starting
0501 13:57:43.102394 (+ 35447us) log.cc:844] Serialized 356 byte log entry
0501 13:57:43.102982 (+ 588us) spinlock_profiling.cc:243] Waited 35 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663024951 00007f56630b7208 00007f56635299f1 00007f56635291c8 00007f566352de61 00007f566352de19 00007f566352dda9 00007f566352dd71 00007f566352cf9d 00000000005b3bc4 00000000005b39f9
0501 13:57:43.561855 (+458873us) op_driver.cc:464] REPLICATION: finished
0501 13:57:43.562291 (+ 436us) write_op.cc:301] APPLY: starting
0501 13:57:43.562346 (+ 55us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:43.562428 (+ 82us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:43.562446 (+ 18us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:43.562627 (+ 181us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:43.562748 (+ 121us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:57:43.562792 (+ 44us) write_op.cc:312] APPLY: finished
0501 13:57:43.562953 (+ 161us) log.cc:844] Serialized 23 byte log entry
0501 13:57:43.563104 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:43.563158 (+ 54us) write_op.cc:454] Released schema lock
0501 13:57:43.563186 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:43.563402 (+ 216us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:43.563425 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":295,"reported_tablets":11,"spinlock_wait_cycles":62925056,"child_traces":[["op",{"apply.queue_time_us":1614,"apply.run_cpu_time_us":1217,"apply.run_wall_time_us":1341,"num_ops":2,"prepare.queue_time_us":74786,"prepare.run_cpu_time_us":1882,"prepare.run_wall_time_us":11379,"replication_time_us":543111}],["op",{"apply.queue_time_us":288,"apply.run_cpu_time_us":1076,"apply.run_wall_time_us":1075,"num_ops":1,"prepare.queue_time_us":188,"prepare.run_cpu_time_us":2610,"prepare.run_wall_time_us":37721,"raft.queue_time_us":378,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":12,"replication_time_us":494805,"spinlock_wait_cycles":80434432}]]}
I20260501 13:57:43.997313 1881 tablet_bootstrap.cc:492] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:43.997726 1881 ts_tablet_manager.cc:1403] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.517s user 0.014s sys 0.011s
I20260501 13:57:43.997620 1757 tablet_service.cc:1511] Processing CreateTablet for tablet 0ff55996178045c0981184330b903bb8 (DEFAULT_TABLE table=default.Table_e2f41fc3135c458e9fda2d60189b4a23 [id=d286fd8e6cb040eaa6f01a571c3e7d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:43.998848 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff55996178045c0981184330b903bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.002537 2526 tablet.cc:1724] T 6650472ec4a449c48c7957a9bbec88e3 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:44.008196 1881 raft_consensus.cc:359] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.008960 1881 raft_consensus.cc:385] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:44.009261 1881 raft_consensus.cc:740] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:44.010016 1881 consensus_queue.cc:260] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.016716 2551 sys_catalog.cc:263] Verifying existing consensus state
I20260501 13:57:44.028110 1881 ts_tablet_manager.cc:1434] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20260501 13:57:44.032955 1883 tablet_bootstrap.cc:492] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:44.039841 1532 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:44.043470 1763 tablet_service.cc:1511] Processing CreateTablet for tablet df1c0b6173654535bd01af7aa3dc6ce9 (DEFAULT_TABLE table=default.Table_31a3f0b22f09450faa99203169288584 [id=6566d5f1bfad40d29ec026d68a935cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:44.043905 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 71ac54444a414e25bcd05452edfbf5ba (DEFAULT_TABLE table=default.Table_60c762a708a54d729a3eb53a841dfe12 [id=2aa7cf25459145adb3f344a5823fba83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:44.044094 1761 tablet_service.cc:1511] Processing CreateTablet for tablet d38d22f2f5b2469a892967b82edbaee5 (DEFAULT_TABLE table=default.Table_79fb3bc412df49ea8ed51d3a8a6f91f5 [id=8cefbb1691bf4ef5852870b1fef2e2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:44.044407 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1c0b6173654535bd01af7aa3dc6ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.044920 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ac54444a414e25bcd05452edfbf5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.045399 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38d22f2f5b2469a892967b82edbaee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.045674 1753 tablet_service.cc:1511] Processing CreateTablet for tablet 4802dafc3e6b4874898b7dee694fc76e (DEFAULT_TABLE table=default.Table_fb6bb57c871e4855aba95f3b430a3fc7 [id=09adc59479f44d358ed186c100b690b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:44.046519 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4802dafc3e6b4874898b7dee694fc76e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.055280 1762 tablet_service.cc:1511] Processing CreateTablet for tablet a000b129a6ab429889b543c20750a493 (DEFAULT_TABLE table=default.Table_31b7fbe1002b47b0adb3e3abb1517807 [id=e2c1080f71804f5b8f68ac9fbcfeb7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:44.056348 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a000b129a6ab429889b543c20750a493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.066315 2551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: Bootstrap starting.
I20260501 13:57:44.068830 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 71f7bd151fa64fe5a80e756a95a375c5 (DEFAULT_TABLE table=default.Table_fb33a28f96da45c29f6e68281ec90b8d [id=1beb747ddd3841f8a489e91ceb05c8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:44.069865 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71f7bd151fa64fe5a80e756a95a375c5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:44.077607 1418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 196129us
I20260501 13:57:44.079130 1779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59508ee6bc6b47959d09f8f5aab89804" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:44.084481 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet eec4bce26de14ea1915faea94490d665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:43 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:44.085701 1756 tablet_service.cc:1511] Processing CreateTablet for tablet bdc412d7b5634ef2893b59467be4fa92 (DEFAULT_TABLE table=default.Table_9f3c5df0faf64dedabec83d1c7d3dd82 [id=606cf51052b443f7aacbd78885edd3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:44.092923 1883 tablet_bootstrap.cc:654] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:44.095463 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdc412d7b5634ef2893b59467be4fa92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.103171 1828 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:44.115273 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet e8efee17e2c34caf98c0ede01d2a943a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:43 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:44.120682 1881 tablet_bootstrap.cc:492] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:44.128433 2564 tablet_replica.cc:333] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:44.129238 2564 raft_consensus.cc:2243] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:44.129688 2564 raft_consensus.cc:2272] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:44.131446 2564 ts_tablet_manager.cc:1916] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:44.139448 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59508ee6bc6b47959d09f8f5aab89804" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:44.142566 2511 leader_election.cc:290] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:44.144049 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 466134us
I20260501 13:57:44.147166 2551 log.cc:826] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: Log is configured to *not* fsync() on all Append() calls
I20260501 13:57:44.156332 1882 tablet_bootstrap.cc:492] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:44.156755 1882 ts_tablet_manager.cc:1403] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.358s user 0.008s sys 0.015s
I20260501 13:57:44.175249 1882 raft_consensus.cc:359] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.175922 1882 raft_consensus.cc:385] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:44.176273 1882 raft_consensus.cc:740] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:44.176378 1779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33a355897309420c97b1257afaefe8d1" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:44.176888 1779 raft_consensus.cc:3060] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:44.176836 1882 consensus_queue.cc:260] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.178702 1882 ts_tablet_manager.cc:1434] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260501 13:57:44.179610 1882 tablet_bootstrap.cc:492] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:44.190264 1882 tablet_bootstrap.cc:654] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:44.198390 2564 ts_tablet_manager.cc:1929] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:44.198864 2564 log.cc:1199] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/e8efee17e2c34caf98c0ede01d2a943a
I20260501 13:57:44.199615 2564 ts_tablet_manager.cc:1950] T e8efee17e2c34caf98c0ede01d2a943a P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:44.208370 1675 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:44.213629 2456 raft_consensus.cc:493] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:44.214126 2456 raft_consensus.cc:515] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.216223 2456 leader_election.cc:290] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:44.217097 1779 raft_consensus.cc:2468] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:44.105317 2047 ts_tablet_manager.cc:1929] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:44.226859 2047 log.cc:1199] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/4f2448ee6bf14bbc8ec0e1d59ccb7b90
I20260501 13:57:44.227623 2047 ts_tablet_manager.cc:1950] T 4f2448ee6bf14bbc8ec0e1d59ccb7b90 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:44.253268 2551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 13:57:44.285998 2551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: Bootstrap complete.
I20260501 13:57:44.298396 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68cb7bd1a5aa4419b9c5176a287fe885" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:44.299969 1717 leader_election.cc:343] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:44.297363 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33a355897309420c97b1257afaefe8d1" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:44.328253 1883 tablet_bootstrap.cc:492] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:44.328687 1883 ts_tablet_manager.cc:1403] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.296s user 0.068s sys 0.075s
I20260501 13:57:44.348193 1882 tablet_bootstrap.cc:492] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:44.348619 1882 ts_tablet_manager.cc:1403] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.169s user 0.009s sys 0.001s
I20260501 13:57:44.337284 1418 leader_election.cc:304] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:44.349558 2536 raft_consensus.cc:2749] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:44.351186 1882 raft_consensus.cc:359] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.351893 1882 raft_consensus.cc:385] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:44.352196 1882 raft_consensus.cc:740] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:44.352816 1882 consensus_queue.cc:260] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:44.334054 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 121033us
I20260501 13:57:44.360294 1882 ts_tablet_manager.cc:1434] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260501 13:57:44.362084 1882 tablet_bootstrap.cc:492] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
W20260501 13:57:44.367058 1119 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.049s sys 0.040s
W20260501 13:57:44.367851 1119 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.617s user 0.050s sys 0.040s
W20260501 13:57:44.368237 1116 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 38c3eb388d474d718554775d65f58d78 (table default.Table_392605e427d14287b4fccdd9b198c6a2 [id=d6ca68cf0d954555b422b90e6502d6cc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 38c3eb388d474d718554775d65f58d78 consensus state has no leader
W20260501 13:57:44.368283 1119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.188:38815, user_credentials={real_user=slave}} blocked reactor thread for 617752us
I20260501 13:57:44.370391 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:44.367873 1117 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 406392us
I20260501 13:57:44.374886 1128 master_service.cc:954] ReplaceTablet: received request to replace tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:44.377262 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 69626ed095bd4df7859563c176dafffc with delete_type TABLET_DATA_DELETED (replaced by tablet 1042bdf58c1f488daeaec284eb3319d5) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:44.378022 2564 tablet_replica.cc:333] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:44.378875 2564 raft_consensus.cc:2243] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:44.379356 2564 raft_consensus.cc:2272] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:44.379501 1125 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_33be6d10669d46efad19a4393e18553d" } modify_external_catalogs: true
I20260501 13:57:44.380409 1125 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_33be6d10669d46efad19a4393e18553d" } modify_external_catalogs: true
I20260501 13:57:44.381615 2564 ts_tablet_manager.cc:1916] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:44.328397 1881 tablet_bootstrap.cc:654] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:44.393816 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 4f2448ee6bf14bbc8ec0e1d59ccb7b90 (table default.Table_041fce4b819d4a4c80c58137b4213234 [id=797990ef44df4a5685fa58bd4c3547bd]) successfully deleted
W20260501 13:57:44.394636 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet e8efee17e2c34caf98c0ede01d2a943a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8efee17e2c34caf98c0ede01d2a943a already in progress: creating tablet
I20260501 13:57:44.396718 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 569a0240b1274d9bb830addbaf1ec343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:44.397792 1131 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_58189c0095284474a78cf7aa10654434" } modify_external_catalogs: true
I20260501 13:57:44.398305 1131 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_58189c0095284474a78cf7aa10654434" } modify_external_catalogs: true
I20260501 13:57:44.400882 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_524cc370fa15424ca5fb7c094c345897"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:44.403494 1130 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_60c762a708a54d729a3eb53a841dfe12" } new_table_name: "default.Table_b69dbcf5f9174f92a4c44217df7dcbcd" modify_external_catalogs: true
I20260501 13:57:44.406522 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_19db51298b7949ebac6642911a90c5c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:44.407110 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet eec4bce26de14ea1915faea94490d665 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eec4bce26de14ea1915faea94490d665 already in progress: creating tablet
W20260501 13:57:44.408185 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19db51298b7949ebac6642911a90c5c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:44.411527 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_524cc370fa15424ca5fb7c094c345897 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:44.435618 1127 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.130:35289 (request call id 70) took 876 ms (client timeout 1000 ms). Trace:
I20260501 13:57:44.436058 1881 tablet_bootstrap.cc:492] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:44.436460 1881 ts_tablet_manager.cc:1403] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.316s user 0.010s sys 0.001s
W20260501 13:57:44.436040 1127 rpcz_store.cc:269] 0501 13:57:43.558867 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:43.566322 (+ 7455us) service_pool.cc:224] Handling call
0501 13:57:44.435432 (+869110us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:44.003171 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:44.003539 (+ 368us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:44.003562 (+ 23us) write_op.cc:435] Acquired schema lock
0501 13:57:44.003574 (+ 12us) tablet.cc:662] Decoding operations
0501 13:57:44.010684 (+ 7110us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:44.010724 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:44.010746 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:57:44.010998 (+ 252us) tablet.cc:701] Row locks acquired
0501 13:57:44.011010 (+ 12us) write_op.cc:260] PREPARE: finished
0501 13:57:44.011126 (+ 116us) write_op.cc:270] Start()
0501 13:57:44.011206 (+ 80us) write_op.cc:276] Timestamp: P: 1777643864011138 usec, L: 0
0501 13:57:44.011221 (+ 15us) op_driver.cc:348] REPLICATION: starting
0501 13:57:44.016269 (+ 5048us) log.cc:844] Serialized 907 byte log entry
0501 13:57:44.017943 (+ 1674us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663024951 00007f56630b7208 00007f56635299f1 00007f56635291c8 00007f566352de61 00007f566352de19 00007f566352dda9 00007f566352dd71 00007f566352cf9d 00000000005b3bc4 00000000005b39f9
0501 13:57:44.411161 (+393218us) op_driver.cc:464] REPLICATION: finished
0501 13:57:44.411646 (+ 485us) write_op.cc:301] APPLY: starting
0501 13:57:44.411741 (+ 95us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:44.411877 (+ 136us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:44.411913 (+ 36us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:44.412327 (+ 414us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:44.412539 (+ 212us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0501 13:57:44.412580 (+ 41us) write_op.cc:312] APPLY: finished
0501 13:57:44.412806 (+ 226us) log.cc:844] Serialized 29 byte log entry
0501 13:57:44.412960 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:44.413060 (+ 100us) write_op.cc:454] Released schema lock
0501 13:57:44.413109 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:44.413421 (+ 312us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:44.413451 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":271,"apply.run_cpu_time_us":1691,"apply.run_wall_time_us":1690,"num_ops":2,"prepare.queue_time_us":110,"prepare.run_cpu_time_us":2631,"prepare.run_wall_time_us":14935,"raft.queue_time_us":144,"raft.run_cpu_time_us":592,"raft.run_wall_time_us":598,"replication_time_us":399790,"spinlock_wait_cycles":10571008}]]}
I20260501 13:57:44.346596 1883 raft_consensus.cc:359] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.451510 1883 raft_consensus.cc:385] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:44.451790 1883 raft_consensus.cc:740] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:44.452379 1883 consensus_queue.cc:260] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.454239 1883 ts_tablet_manager.cc:1434] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.125s user 0.000s sys 0.004s
I20260501 13:57:44.462682 1881 raft_consensus.cc:359] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.463302 1881 raft_consensus.cc:385] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:44.455078 1883 tablet_bootstrap.cc:492] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:44.463557 1881 raft_consensus.cc:740] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:44.464511 1881 consensus_queue.cc:260] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.468890 1883 tablet_bootstrap.cc:654] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:44.473541 2564 ts_tablet_manager.cc:1929] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:44.473923 2564 log.cc:1199] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/a2cf9528d5c24fc68e3dfdd93bf5c7bb
I20260501 13:57:44.474607 2564 ts_tablet_manager.cc:1950] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:44.477561 2564 tablet_replica.cc:333] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:44.478341 2564 raft_consensus.cc:2243] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:44.478818 2564 raft_consensus.cc:2272] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:44.480561 2564 ts_tablet_manager.cc:1916] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:44.360915 1563 leader_election.cc:304] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
W20260501 13:57:44.508085 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 147107us
I20260501 13:57:44.439059 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ef8d1ad25b3a469cb3d3840b998178d8" } new_table_name: "default.Table_511cd4142c9642d98428697731daef7a" modify_external_catalogs: true
I20260501 13:57:44.440423 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_ca507f190ca9413d91939e27911c79d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:44.429257 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet eec4bce26de14ea1915faea94490d665 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eec4bce26de14ea1915faea94490d665 already in progress: creating tablet
W20260501 13:57:44.511819 1116 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 82604.9us
W20260501 13:57:44.512792 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca507f190ca9413d91939e27911c79d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:44.513367 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet e8efee17e2c34caf98c0ede01d2a943a (table default.Table_ce141ed4961d49a8a07433b1dded9ce5 [id=cf2193b6ac264ef290b4bfb40970f215]) successfully deleted
W20260501 13:57:44.513530 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 372037us
I20260501 13:57:44.513990 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb (table default.Table_e7f17dee34594eb3a711ffbf706d4e59 [id=8570b73bebda4640a91c1bc894942e38]) successfully deleted
I20260501 13:57:44.461882 1125 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 53d9a8503c8d4bebb89ea493b99bf6fd
I20260501 13:57:44.466300 1881 ts_tablet_manager.cc:1434] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
W20260501 13:57:44.491925 1130 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71ac54444a414e25bcd05452edfbf5ba consensus state has no leader
I20260501 13:57:44.517112 1881 tablet_bootstrap.cc:492] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:44.519807 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 53d9a8503c8d4bebb89ea493b99bf6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:44 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:44.524086 1480 raft_consensus.cc:3060] T 59508ee6bc6b47959d09f8f5aab89804 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:44.531068 1564 leader_election.cc:343] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:44.531386 1456 tablet_service.cc:1558] Processing DeleteTablet for tablet 53d9a8503c8d4bebb89ea493b99bf6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:44 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:44.535327 1881 tablet_bootstrap.cc:654] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:44.540974 2456 raft_consensus.cc:493] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:44.541474 2456 raft_consensus.cc:515] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.543462 2456 leader_election.cc:290] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
W20260501 13:57:44.549379 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 53d9a8503c8d4bebb89ea493b99bf6fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53d9a8503c8d4bebb89ea493b99bf6fd already in progress: creating tablet
W20260501 13:57:44.556201 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62d7ad7aa5ec4f79a2294023e17766d2 consensus state has no leader
I20260501 13:57:44.559919 1601 tablet_service.cc:1511] Processing CreateTablet for tablet 71f7bd151fa64fe5a80e756a95a375c5 (DEFAULT_TABLE table=default.Table_fb33a28f96da45c29f6e68281ec90b8d [id=1beb747ddd3841f8a489e91ceb05c8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:44.561748 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71f7bd151fa64fe5a80e756a95a375c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.562875 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet a9672aab85c64d548a4f6583a1e9af20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:44.571241 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet 863a10cfd14c4694b7c6141c0851167a with delete_type TABLET_DATA_DELETED (replaced by tablet 6f6cd353ca044465a6692656412e7d82) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:44.571563 1607 tablet_service.cc:1558] Processing DeleteTablet for tablet eec4bce26de14ea1915faea94490d665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:43 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:44.572129 1608 tablet_service.cc:1558] Processing DeleteTablet for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:44.571242 1606 tablet_service.cc:1558] Processing DeleteTablet for tablet e8efee17e2c34caf98c0ede01d2a943a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:43 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:44.580535 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet e0356b19f0aa4197a41cc66d4440bffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:44.588028 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4a285dc47b4c471792aecb0dd81531ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:44.589923 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a285dc47b4c471792aecb0dd81531ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:44.591545 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68cb7bd1a5aa4419b9c5176a287fe885" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:44.560912 1532 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:44.582049 2551 raft_consensus.cc:359] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:44.609665 2551 raft_consensus.cc:740] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c0932256e74399a537670bb424a1b7, State: Initialized, Role: FOLLOWER
I20260501 13:57:44.612663 2551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:44.626709 1882 tablet_bootstrap.cc:654] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:44.635501 1609 tablet_service.cc:1511] Processing CreateTablet for tablet bdc412d7b5634ef2893b59467be4fa92 (DEFAULT_TABLE table=default.Table_9f3c5df0faf64dedabec83d1c7d3dd82 [id=606cf51052b443f7aacbd78885edd3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:44.636880 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdc412d7b5634ef2893b59467be4fa92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.639047 1883 tablet_bootstrap.cc:492] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:44.639467 1883 ts_tablet_manager.cc:1403] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.185s user 0.016s sys 0.008s
I20260501 13:57:44.640120 2499 raft_consensus.cc:2804] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:44.641836 1883 raft_consensus.cc:359] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.641716 2302 consensus_queue.cc:1048] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:57:44.653394 1828 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:44.657299 2536 raft_consensus.cc:493] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:44.658179 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2ac522f2e4c4ab48a9b7d211c82ed9e" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:44.642407 1883 raft_consensus.cc:385] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:44.680806 1883 raft_consensus.cc:740] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:44.681635 2564 ts_tablet_manager.cc:1929] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:44.682016 2564 log.cc:1199] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/69626ed095bd4df7859563c176dafffc
I20260501 13:57:44.683799 1883 consensus_queue.cc:260] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.685767 2564 ts_tablet_manager.cc:1950] T 69626ed095bd4df7859563c176dafffc P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:44.687125 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 69626ed095bd4df7859563c176dafffc (table default.Table_0c2d991875174eefbc2fa69108cc7082 [id=63f1fe234b1143869f21b76118b58234]) successfully deleted
I20260501 13:57:44.679265 2536 raft_consensus.cc:515] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.691912 2564 tablet_replica.cc:333] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:44.692493 2536 leader_election.cc:290] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:44.692760 2564 raft_consensus.cc:2243] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:44.694212 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eb6d30bed2a410db8b62fdff54a5fac" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
W20260501 13:57:44.694249 1717 leader_election.cc:343] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:44.698292 1883 ts_tablet_manager.cc:1434] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.058s user 0.005s sys 0.000s
I20260501 13:57:44.698225 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_295132c6928c4b309137696f9015de9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:44.699163 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_b1d9b08159ee48c7bb3e453da9f6ff40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:44.701318 1883 tablet_bootstrap.cc:492] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:44.706943 2564 raft_consensus.cc:2272] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:44.710628 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1d9b08159ee48c7bb3e453da9f6ff40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:44.712416 2564 ts_tablet_manager.cc:1916] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:44.713975 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_295132c6928c4b309137696f9015de9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:44.718343 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_407a6cfd065142f4a2debee031077ab9" } modify_external_catalogs: true
I20260501 13:57:44.718932 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_407a6cfd065142f4a2debee031077ab9" } modify_external_catalogs: true
I20260501 13:57:44.719844 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_3f7be42c55ff49c0ac1a846980e70e15" } new_table_name: "default.Table_86cb74d55d764d4abd4f2542b889ecf2" modify_external_catalogs: true
I20260501 13:57:44.724875 1883 tablet_bootstrap.cc:654] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:44.746294 1125 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 665a8d647e494b44b17ad6cdb02337c8 has no consensus state
I20260501 13:57:44.747802 2583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:44.748967 2583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:44.754989 2582 tablet_replica.cc:333] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:44.759342 2551 sys_catalog.cc:565] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 13:57:44.761463 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_d547fe7eda17499384b316b482c599f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:44.762993 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d547fe7eda17499384b316b482c599f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:44.764798 2564 ts_tablet_manager.cc:1929] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:44.765177 2564 log.cc:1199] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/569a0240b1274d9bb830addbaf1ec343
I20260501 13:57:44.765812 2564 ts_tablet_manager.cc:1950] T 569a0240b1274d9bb830addbaf1ec343 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:44.771240 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 569a0240b1274d9bb830addbaf1ec343 (table default.Table_c9c10d0b01f846849dcd931a75ff0d63 [id=118a5cac4b274ea8879ae4b0dc467be4]) successfully deleted
W20260501 13:57:44.758317 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 224777us
I20260501 13:57:44.772209 1881 tablet_bootstrap.cc:492] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:44.772703 1881 ts_tablet_manager.cc:1403] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.256s user 0.013s sys 0.006s
W20260501 13:57:44.773387 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 53d9a8503c8d4bebb89ea493b99bf6fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53d9a8503c8d4bebb89ea493b99bf6fd already in progress: creating tablet
I20260501 13:57:44.749840 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eb6d30bed2a410db8b62fdff54a5fac" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:44.775170 1881 raft_consensus.cc:359] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.784341 1883 tablet_bootstrap.cc:492] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:44.784858 1883 ts_tablet_manager.cc:1403] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.084s user 0.014s sys 0.012s
I20260501 13:57:44.786268 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6ed6c8b736e9485789f83a5b49e1d994" } new_table_name: "default.Table_ca91828e5c8642c3be033ef95e53f2a3" modify_external_catalogs: true
I20260501 13:57:44.786903 1881 raft_consensus.cc:385] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:44.787246 1881 raft_consensus.cc:740] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:44.788133 1881 consensus_queue.cc:260] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.801441 1123 master_service.cc:954] ReplaceTablet: received request to replace tablet 62d7ad7aa5ec4f79a2294023e17766d2 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:44.804045 1130 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 96e937b3a0b5451e8eaf4d5c47c17620 (table default.Table_b29786a52abd4fad9a2337d7c76fbbc4 [id=05998ead56ed454dab9a71ef2ac9cb58]). Expected version 1 got 0
I20260501 13:57:44.804764 1130 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715]). Expected version 1 got 0
I20260501 13:57:44.750075 2499 raft_consensus.cc:697] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:44.813256 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 7d399de014d74cab8df14f44787ca2f7 (table default.Table_aa47824eda84455d9d52df61514a253e [id=81117c514ffd4ab6a8d4dde20d7559e4]). Expected version 1 got 0
I20260501 13:57:44.813936 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 1409fbe5846a4bb099f31fb2d2e41382 (table default.Table_be2275f0abfb43abb3bd0b1e128d7fef [id=8d39f1f13bc24a2c84e888bb29b6e74c]). Expected version 1 got 0
I20260501 13:57:44.814718 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715]). Expected version 1 got 0
I20260501 13:57:44.817655 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_151d21fe56664c948925ce791cd0a061"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:44.819268 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_151d21fe56664c948925ce791cd0a061 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:44.811035 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 60395.4us
I20260501 13:57:44.817611 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 56cee07054a64e199a0bfc26b63eca2a (table default.Table_9b824b59406d478884838d29a5d690f8 [id=389132d3db474d008cc0ee9d758b4fec]). Expected version 1 got 0
I20260501 13:57:44.822140 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 96e937b3a0b5451e8eaf4d5c47c17620 (table default.Table_b29786a52abd4fad9a2337d7c76fbbc4 [id=05998ead56ed454dab9a71ef2ac9cb58]). Expected version 1 got 0
I20260501 13:57:44.821583 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 75fe098e6579473fadf371aea2c001b8 (DEFAULT_TABLE table=default.Table_b12841aa2a8240a0b6e3523a0e583b98 [id=227b30167bab4a68ad46c1d05213b2f3]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:44.817332 2344 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.009s sys 0.040s
W20260501 13:57:44.823652 2344 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s user 0.009s sys 0.040s
I20260501 13:57:44.823760 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75fe098e6579473fadf371aea2c001b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.824273 1881 ts_tablet_manager.cc:1434] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.051s user 0.004s sys 0.000s
I20260501 13:57:44.825116 1881 tablet_bootstrap.cc:492] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:44.827200 1883 raft_consensus.cc:359] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.827800 1883 raft_consensus.cc:385] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:44.828037 1883 raft_consensus.cc:740] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
W20260501 13:57:44.826377 2571 log.cc:927] Time spent T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20260501 13:57:44.828603 1883 consensus_queue.cc:260] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.833892 1882 tablet_bootstrap.cc:492] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:44.834338 1882 ts_tablet_manager.cc:1403] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.472s user 0.016s sys 0.000s
I20260501 13:57:44.843559 2582 raft_consensus.cc:2243] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:44.844233 2582 raft_consensus.cc:2272] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:44.844735 1421 leader_election.cc:304] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:44.845750 2536 raft_consensus.cc:2749] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:44.846268 2582 ts_tablet_manager.cc:1916] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:44.861136 2499 consensus_queue.cc:237] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.885227 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 75fe098e6579473fadf371aea2c001b8 (DEFAULT_TABLE table=default.Table_b12841aa2a8240a0b6e3523a0e583b98 [id=227b30167bab4a68ad46c1d05213b2f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:44.886241 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75fe098e6579473fadf371aea2c001b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.893951 1883 ts_tablet_manager.cc:1434] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20260501 13:57:44.894968 1883 tablet_bootstrap.cc:492] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:44.903749 1882 raft_consensus.cc:359] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.904408 1882 raft_consensus.cc:385] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:44.904665 1882 raft_consensus.cc:740] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
W20260501 13:57:44.904961 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29f0e9def65d4ba9ad5aded73ea20515 consensus state has no leader
I20260501 13:57:44.905210 1882 consensus_queue.cc:260] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:44.906649 1131 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 2b7e55c79aa74ee6b164758daf6325fa
I20260501 13:57:44.908790 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b7e55c79aa74ee6b164758daf6325fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:44 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:44.909994 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b7e55c79aa74ee6b164758daf6325fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:44 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:44.911425 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 2b7e55c79aa74ee6b164758daf6325fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b7e55c79aa74ee6b164758daf6325fa already in progress: creating tablet
W20260501 13:57:44.911561 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 2b7e55c79aa74ee6b164758daf6325fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b7e55c79aa74ee6b164758daf6325fa already in progress: creating tablet
I20260501 13:57:44.922966 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 2a5f2c9b3e624fcebd2ddc86d66b08c4 (DEFAULT_TABLE table=default.Table_9bd7fbc81aa64971ab1296f32b26179f [id=5c4a71ef127542c99ed8d77b8eea402a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:44.924014 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a5f2c9b3e624fcebd2ddc86d66b08c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:44.928004 1127 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 3b0ed592fd3642c194bb946c8c6003b4
I20260501 13:57:44.931087 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b0ed592fd3642c194bb946c8c6003b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:44 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:44.932072 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b0ed592fd3642c194bb946c8c6003b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:44 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:44.933005 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 3b0ed592fd3642c194bb946c8c6003b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b0ed592fd3642c194bb946c8c6003b4 already in progress: creating tablet
W20260501 13:57:44.933609 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 3b0ed592fd3642c194bb946c8c6003b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b0ed592fd3642c194bb946c8c6003b4 already in progress: creating tablet
I20260501 13:57:44.934263 1123 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 62d7ad7aa5ec4f79a2294023e17766d2
I20260501 13:57:44.935482 1123 catalog_manager.cc:6741] ReplaceTablet: tablet 62d7ad7aa5ec4f79a2294023e17766d2 deleted and replaced by tablet 6c7e65dad084440787754e24cdd579d9
I20260501 13:57:44.935910 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet 62d7ad7aa5ec4f79a2294023e17766d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c7e65dad084440787754e24cdd579d9) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:44.937615 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 62d7ad7aa5ec4f79a2294023e17766d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62d7ad7aa5ec4f79a2294023e17766d2 already in progress: creating tablet
I20260501 13:57:44.938293 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet 62d7ad7aa5ec4f79a2294023e17766d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c7e65dad084440787754e24cdd579d9) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:44.939709 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 62d7ad7aa5ec4f79a2294023e17766d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62d7ad7aa5ec4f79a2294023e17766d2 already in progress: creating tablet
I20260501 13:57:44.939958 1881 tablet_bootstrap.cc:654] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:44.944459 1130 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_0d92e7b95c884bd5970e291f4189b0d9 [id=774a7eb90ba449b6aef225d819678439] alter complete (version 1)
I20260501 13:57:44.954543 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_56f3ff0cf6ff4fe4974017f9ed722ca5" } new_table_name: "default.Table_6fe907773720414d8f8301eb3839c456" modify_external_catalogs: true
I20260501 13:57:44.954672 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_44c278d4bf6c4c85b644cc7b115ce1e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:44.956290 1883 tablet_bootstrap.cc:654] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:44.956378 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44c278d4bf6c4c85b644cc7b115ce1e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:44.958159 1882 ts_tablet_manager.cc:1434] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.124s user 0.006s sys 0.000s
I20260501 13:57:44.959187 1882 tablet_bootstrap.cc:492] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
W20260501 13:57:44.963598 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 158451us
W20260501 13:57:44.985503 1719 leader_election.cc:343] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:44.985921 1719 leader_election.cc:304] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:44.986593 2456 raft_consensus.cc:2749] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:44.995467 1130 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_ecd46a159d2448f9ba09fc99e14a22a3 [id=d98d5437c5bc4e44bcfe27983dfa36e6] alter complete (version 1)
I20260501 13:57:45.005339 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet 69626ed095bd4df7859563c176dafffc with delete_type TABLET_DATA_DELETED (replaced by tablet 1042bdf58c1f488daeaec284eb3319d5) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:45.007189 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:45.008970 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 68cb7bd1a5aa4419b9c5176a287fe885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:45.009582 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet 569a0240b1274d9bb830addbaf1ec343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:45.011569 1756 tablet_service.cc:1511] Processing CreateTablet for tablet ddeccfad2eb749bbbacf0a4543fff970 (DEFAULT_TABLE table=default.Table_7fdfc794bcaf49488caa439514624cec [id=e9a43884545843c6ad1fa78f28173910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.012619 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 2a5f2c9b3e624fcebd2ddc86d66b08c4 (DEFAULT_TABLE table=default.Table_9bd7fbc81aa64971ab1296f32b26179f [id=5c4a71ef127542c99ed8d77b8eea402a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.012562 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_1a37845ff6b34f87ae7b12cbc4738328"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:45.013978 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a37845ff6b34f87ae7b12cbc4738328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:45.016500 1455 tablet_service.cc:1511] Processing CreateTablet for tablet ddeccfad2eb749bbbacf0a4543fff970 (DEFAULT_TABLE table=default.Table_7fdfc794bcaf49488caa439514624cec [id=e9a43884545843c6ad1fa78f28173910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.017639 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddeccfad2eb749bbbacf0a4543fff970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.018698 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet b98187c7843e42b09d55c851117c4ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:45.018929 1125 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef622e34d46a4b19a59e916d22c2b18b consensus state has no leader
I20260501 13:57:45.019699 1464 tablet_service.cc:1511] Processing CreateTablet for tablet f260f3e32b944d8ba1b1081cf43a12ac (DEFAULT_TABLE table=default.Table_7ff8525575ef446aad5a61795d4018a7 [id=bdb8d8c4a0514c37a664c2108a60a98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.021018 2047 tablet_replica.cc:333] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:45.021766 2047 raft_consensus.cc:2243] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:45.022265 2047 raft_consensus.cc:2272] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:45.022295 1754 tablet_service.cc:1511] Processing CreateTablet for tablet f260f3e32b944d8ba1b1081cf43a12ac (DEFAULT_TABLE table=default.Table_7ff8525575ef446aad5a61795d4018a7 [id=bdb8d8c4a0514c37a664c2108a60a98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.023964 2047 ts_tablet_manager.cc:1916] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:45.027974 1465 tablet_service.cc:1511] Processing CreateTablet for tablet 3d0fa1a12a554984a893ec75e8b32382 (DEFAULT_TABLE table=default.Table_a308b6848f114c1ead6ecf2798c61d37 [id=ad9937f10bdb410fb9d08ecc8746e4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.030706 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f260f3e32b944d8ba1b1081cf43a12ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.031237 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d0fa1a12a554984a893ec75e8b32382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.040349 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 3d0fa1a12a554984a893ec75e8b32382 (DEFAULT_TABLE table=default.Table_a308b6848f114c1ead6ecf2798c61d37 [id=ad9937f10bdb410fb9d08ecc8746e4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.043879 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet ec4bd4cb1e70472aa09fc93afc3b1501 with delete_type TABLET_DATA_DELETED (replaced by tablet 8dbb619863dd405ab20ead6bbebf4d69) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:45.044066 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet 53d9a8503c8d4bebb89ea493b99bf6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:44 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:45.044526 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 0669a229d6d54eacac9889ad18e06f30 with delete_type TABLET_DATA_DELETED (replaced by tablet 81230beeff8d434aa2dac467c35d9703) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:45.045411 2590 raft_consensus.cc:493] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:45.046041 2590 raft_consensus.cc:515] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.047060 2580 raft_consensus.cc:493] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:45.047469 2580 raft_consensus.cc:515] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.049594 2580 leader_election.cc:290] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:45.053272 1610 tablet_service.cc:1511] Processing CreateTablet for tablet 75fe098e6579473fadf371aea2c001b8 (DEFAULT_TABLE table=default.Table_b12841aa2a8240a0b6e3523a0e583b98 [id=227b30167bab4a68ad46c1d05213b2f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.053833 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet a4265d9ab3734a8b97d745dc02c6b845 (table default.Table_4440a2fc30004b389edb91265721ba86 [id=bd84d43e8e074a9383c9843cd13e7a65]). Expected version 1 got 0
I20260501 13:57:45.054265 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75fe098e6579473fadf371aea2c001b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.054502 1480 raft_consensus.cc:3060] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:45.060585 1563 leader_election.cc:343] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: e1d7a80ad4c4455a9ddee7a955e32635
I20260501 13:57:45.062923 1125 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715]). Expected version 1 got 0
I20260501 13:57:45.063560 1125 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 96e937b3a0b5451e8eaf4d5c47c17620 (table default.Table_b29786a52abd4fad9a2337d7c76fbbc4 [id=05998ead56ed454dab9a71ef2ac9cb58]). Expected version 1 got 0
I20260501 13:57:45.064816 2590 leader_election.cc:290] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:45.070751 1563 leader_election.cc:343] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 67b0675398d04be08bd29638b4ba2083
I20260501 13:57:45.072798 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 74af56cca54347bab67895e10ee17230 (DEFAULT_TABLE table=default.Table_8a2d4f67afd1427b8c32c5bb223a7148 [id=4668751f7bc649019547c38d0f8254c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.080915 2607 tablet.cc:1724] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:45.090828 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b7e55c79aa74ee6b164758daf6325fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:44 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:45.096022 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 74af56cca54347bab67895e10ee17230 (DEFAULT_TABLE table=default.Table_8a2d4f67afd1427b8c32c5bb223a7148 [id=4668751f7bc649019547c38d0f8254c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.097069 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74af56cca54347bab67895e10ee17230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.098165 2047 ts_tablet_manager.cc:1929] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:45.102442 2536 raft_consensus.cc:493] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:45.103087 2536 raft_consensus.cc:515] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.105221 2536 leader_election.cc:290] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:45.105976 2582 ts_tablet_manager.cc:1929] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:45.106323 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_769d5d15b713487f9183c5bc67155757"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:45.108271 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_769d5d15b713487f9183c5bc67155757 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:45.108448 1882 tablet_bootstrap.cc:654] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:45.108982 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_de443751750240b6a298d8aca41e6b56" } new_table_name: "default.Table_774952a38792463085e33c49bf79f603" modify_external_catalogs: true
I20260501 13:57:45.111611 1124 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_b69dbcf5f9174f92a4c44217df7dcbcd" } modify_external_catalogs: true
I20260501 13:57:45.111981 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 29f0e9def65d4ba9ad5aded73ea20515 (DEFAULT_TABLE table=default.Table_ca91828e5c8642c3be033ef95e53f2a3 [id=b733e10eaf5a438ca21057e62556588a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.112996 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f0e9def65d4ba9ad5aded73ea20515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.114789 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b0ed592fd3642c194bb946c8c6003b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:44 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:45.120023 2564 tablet_replica.cc:333] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:45.121012 2564 raft_consensus.cc:2243] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:45.121510 2564 raft_consensus.cc:2272] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:45.126432 2564 ts_tablet_manager.cc:1916] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:45.131917 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6fe907773720414d8f8301eb3839c456" } modify_external_catalogs: true
I20260501 13:57:45.138082 1602 tablet_service.cc:1511] Processing CreateTablet for tablet 2a5f2c9b3e624fcebd2ddc86d66b08c4 (DEFAULT_TABLE table=default.Table_9bd7fbc81aa64971ab1296f32b26179f [id=5c4a71ef127542c99ed8d77b8eea402a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.139163 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a5f2c9b3e624fcebd2ddc86d66b08c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.147022 1132 master_service.cc:954] ReplaceTablet: received request to replace tablet 53d9a8503c8d4bebb89ea493b99bf6fd from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:45.147783 1605 tablet_service.cc:1511] Processing CreateTablet for tablet f260f3e32b944d8ba1b1081cf43a12ac (DEFAULT_TABLE table=default.Table_7ff8525575ef446aad5a61795d4018a7 [id=bdb8d8c4a0514c37a664c2108a60a98a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.148197 1601 tablet_service.cc:1511] Processing CreateTablet for tablet ddeccfad2eb749bbbacf0a4543fff970 (DEFAULT_TABLE table=default.Table_7fdfc794bcaf49488caa439514624cec [id=e9a43884545843c6ad1fa78f28173910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.148787 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f260f3e32b944d8ba1b1081cf43a12ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.149506 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddeccfad2eb749bbbacf0a4543fff970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.152052 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_ecb1d6e04f584898ae14b3aec89681a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:45.153553 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecb1d6e04f584898ae14b3aec89681a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:45.106392 2582 log.cc:1199] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/a9672aab85c64d548a4f6583a1e9af20
I20260501 13:57:45.159294 2582 ts_tablet_manager.cc:1950] T a9672aab85c64d548a4f6583a1e9af20 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:45.161494 2582 tablet_replica.cc:333] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:45.162675 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 62d7ad7aa5ec4f79a2294023e17766d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c7e65dad084440787754e24cdd579d9) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:45.167765 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd02738e5f4ecfbfa06d5541614ffd" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:45.168532 1627 raft_consensus.cc:2468] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:45.173918 1421 leader_election.cc:304] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:45.180433 1883 tablet_bootstrap.cc:492] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:45.180938 1883 ts_tablet_manager.cc:1403] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.286s user 0.009s sys 0.003s
I20260501 13:57:45.175873 1881 tablet_bootstrap.cc:492] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:45.186861 1881 ts_tablet_manager.cc:1403] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.362s user 0.009s sys 0.002s
I20260501 13:57:45.189230 1881 raft_consensus.cc:359] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.189894 1881 raft_consensus.cc:385] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:45.190171 1881 raft_consensus.cc:740] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:45.190973 1881 consensus_queue.cc:260] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.194938 2536 raft_consensus.cc:2804] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:45.195309 2536 raft_consensus.cc:493] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:45.195673 2536 raft_consensus.cc:3060] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:45.203593 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_6fe907773720414d8f8301eb3839c456" } modify_external_catalogs: true
I20260501 13:57:45.204649 2536 raft_consensus.cc:515] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.206820 2536 leader_election.cc:290] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:45.209168 1124 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_b69dbcf5f9174f92a4c44217df7dcbcd" } modify_external_catalogs: true
I20260501 13:57:45.183336 1883 raft_consensus.cc:359] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.209883 1883 raft_consensus.cc:385] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:45.210278 1883 raft_consensus.cc:740] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:45.212338 1883 consensus_queue.cc:260] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.162329 2582 raft_consensus.cc:2243] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:45.054447 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 406e1af8b7d9488f88697206daa87c05 (table default.Table_759f3009542147de94c73784e1698246 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]). Expected version 1 got 0
I20260501 13:57:45.230940 1601 tablet_service.cc:1511] Processing CreateTablet for tablet 3d0fa1a12a554984a893ec75e8b32382 (DEFAULT_TABLE table=default.Table_a308b6848f114c1ead6ecf2798c61d37 [id=ad9937f10bdb410fb9d08ecc8746e4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.219168 1881 ts_tablet_manager.cc:1434] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260501 13:57:45.235643 1881 tablet_bootstrap.cc:492] T 406e1af8b7d9488f88697206daa87c05 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:45.219498 2047 log.cc:1199] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/b98187c7843e42b09d55c851117c4ac8
I20260501 13:57:45.239427 2047 ts_tablet_manager.cc:1950] T b98187c7843e42b09d55c851117c4ac8 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:45.224479 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd02738e5f4ecfbfa06d5541614ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:45.240291 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 29f0e9def65d4ba9ad5aded73ea20515 (DEFAULT_TABLE table=default.Table_ca91828e5c8642c3be033ef95e53f2a3 [id=b733e10eaf5a438ca21057e62556588a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.224993 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet a9672aab85c64d548a4f6583a1e9af20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:45.240845 1610 tablet_service.cc:1511] Processing CreateTablet for tablet 74af56cca54347bab67895e10ee17230 (DEFAULT_TABLE table=default.Table_8a2d4f67afd1427b8c32c5bb223a7148 [id=4668751f7bc649019547c38d0f8254c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.241763 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet b98187c7843e42b09d55c851117c4ac8 (table default.Table_8999f52c517a462a9845800aba73657e [id=e1549f4369a14aceb962e2be6f1a2ea4]) successfully deleted
I20260501 13:57:45.242295 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet a9672aab85c64d548a4f6583a1e9af20 (table default.Table_bc1aee595536420799f8a28881253c09 [id=787286aa1fb44c45827992c1cf240e2d]) successfully deleted
W20260501 13:57:45.243201 1119 catalog_manager.cc:4969] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 863a10cfd14c4694b7c6141c0851167a because the tablet was not found. No further retry: Not found: Tablet not found: 863a10cfd14c4694b7c6141c0851167a
W20260501 13:57:45.244050 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet eec4bce26de14ea1915faea94490d665 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eec4bce26de14ea1915faea94490d665 already in progress: creating tablet
I20260501 13:57:45.248826 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d7a80ad4c4455a9ddee7a955e32635" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:45.251837 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67b0675398d04be08bd29638b4ba2083" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:45.253933 2047 tablet_replica.cc:333] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:45.255357 2047 raft_consensus.cc:2243] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:45.255494 2580 raft_consensus.cc:493] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:45.255954 2580 raft_consensus.cc:515] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.256157 2047 raft_consensus.cc:2272] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:45.258204 2580 leader_election.cc:290] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:45.258810 2582 raft_consensus.cc:2272] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:45.270206 1564 leader_election.cc:343] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.271123 1564 leader_election.cc:304] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:45.274943 2520 raft_consensus.cc:2749] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:45.276242 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c22c2f8e55a4be99a8c6ff32bace441 consensus state has no leader
I20260501 13:57:45.260936 2582 ts_tablet_manager.cc:1916] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:45.284622 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_22d1c2cee41e4e9dad9e52bde00f1524" } new_table_name: "default.Table_14a392c71f144eb0a4e0d5ac408e98e9" modify_external_catalogs: true
I20260501 13:57:45.289229 2047 ts_tablet_manager.cc:1916] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:45.295035 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd02738e5f4ecfbfa06d5541614ffd" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:45.295650 1629 raft_consensus.cc:3060] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:45.303023 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2ac522f2e4c4ab48a9b7d211c82ed9e" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:45.312871 1629 raft_consensus.cc:2468] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:45.319052 1421 leader_election.cc:304] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
W20260501 13:57:45.319832 1421 raft_consensus.cc:2686] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260501 13:57:45.333012 2610 tablet.cc:1724] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:45.335273 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet e0356b19f0aa4197a41cc66d4440bffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:45.334682 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d0fa1a12a554984a893ec75e8b32382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.335667 1882 tablet_bootstrap.cc:492] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:45.344419 1882 ts_tablet_manager.cc:1403] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.385s user 0.019s sys 0.002s
I20260501 13:57:45.344655 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f0e9def65d4ba9ad5aded73ea20515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.345589 1881 tablet_bootstrap.cc:654] T 406e1af8b7d9488f88697206daa87c05 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:45.346700 1882 raft_consensus.cc:359] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.347322 1882 raft_consensus.cc:385] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:45.347556 1882 raft_consensus.cc:740] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:45.348080 1882 consensus_queue.cc:260] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.348575 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74af56cca54347bab67895e10ee17230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.349884 1882 ts_tablet_manager.cc:1434] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260501 13:57:45.356994 2536 raft_consensus.cc:493] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:45.357522 2536 raft_consensus.cc:515] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.359805 2536 leader_election.cc:290] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:45.361977 1421 leader_election.cc:343] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Not found: Tablet not found: a9672aab85c64d548a4f6583a1e9af20
I20260501 13:57:45.364707 1882 tablet_bootstrap.cc:492] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:45.368762 1881 tablet_bootstrap.cc:492] T 406e1af8b7d9488f88697206daa87c05 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:45.369153 1881 ts_tablet_manager.cc:1403] T 406e1af8b7d9488f88697206daa87c05 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.134s user 0.004s sys 0.005s
I20260501 13:57:45.369652 1882 tablet_bootstrap.cc:654] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:45.336071 2611 tablet.cc:1724] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 13:57:45.337069 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 66947.7us
W20260501 13:57:45.387741 1564 leader_election.cc:343] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.389508 1564 leader_election.cc:304] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:45.390134 2520 raft_consensus.cc:2749] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:45.400875 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_bc1658a0432141c1b6e439a4e36ffc33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:45.402770 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d361a2d82ecd456fb43f5e7fe3c83094" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:45.402335 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc1658a0432141c1b6e439a4e36ffc33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:45.405887 1118 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 2c22c2f8e55a4be99a8c6ff32bace441 (table default.Table_774952a38792463085e33c49bf79f603 [id=d1cfd215cd17431ba79e6d2645eea53f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2c22c2f8e55a4be99a8c6ff32bace441 consensus state has no leader
W20260501 13:57:45.442642 1564 leader_election.cc:343] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.447324 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 71ac54444a414e25bcd05452edfbf5ba
I20260501 13:57:45.449100 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet ef622e34d46a4b19a59e916d22c2b18b
I20260501 13:57:45.448931 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet 71ac54444a414e25bcd05452edfbf5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:45.453037 2623 raft_consensus.cc:493] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:45.453845 2623 raft_consensus.cc:515] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.456686 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet ef622e34d46a4b19a59e916d22c2b18b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:45.457325 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet 71ac54444a414e25bcd05452edfbf5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:45.459021 1125 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9 consensus state has no leader
I20260501 13:57:45.462219 2624 raft_consensus.cc:493] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:45.467900 1882 tablet_bootstrap.cc:492] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:45.468338 1882 ts_tablet_manager.cc:1403] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.104s user 0.010s sys 0.007s
I20260501 13:57:45.468214 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_aab8dd1a3e894677b333dc115aa2222c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:45.478332 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aab8dd1a3e894677b333dc115aa2222c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:45.479516 1882 raft_consensus.cc:359] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.479954 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_96180f484b044c0b85bb2f5d8a14d194" } new_table_name: "default.Table_64024529e14042c3be91a36b033ada90" modify_external_catalogs: true
I20260501 13:57:45.480137 1882 raft_consensus.cc:385] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:45.480382 1882 raft_consensus.cc:740] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:45.481817 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9169851f1358460d99a48cb6067ab3e3" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:45.484066 1882 consensus_queue.cc:260] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.493750 1882 ts_tablet_manager.cc:1434] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20260501 13:57:45.497818 1882 tablet_bootstrap.cc:492] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:45.502673 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet ef622e34d46a4b19a59e916d22c2b18b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:45.507104 2631 tablet.cc:1724] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:45.507449 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_51d3bb23841a45c78ff5cc7e492e0c14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:45.509882 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 69626ed095bd4df7859563c176dafffc because the tablet was not found. No further retry: Not found: Tablet not found: 69626ed095bd4df7859563c176dafffc
W20260501 13:57:45.511050 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb because the tablet was not found. No further retry: Not found: Tablet not found: a2cf9528d5c24fc68e3dfdd93bf5c7bb
I20260501 13:57:45.514989 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9169851f1358460d99a48cb6067ab3e3" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:45.474872 2623 leader_election.cc:290] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:45.477160 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f260f3e32b944d8ba1b1081cf43a12ac. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:45.483381 1717 leader_election.cc:343] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.525075 2610 tablet.cc:1724] T ffe89d656956440e8d0c1de0b01ce486 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:45.530777 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_1f1a1e3878d641fc88bcb55193ac5d67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:45.532239 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f1a1e3878d641fc88bcb55193ac5d67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:45.538511 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51d3bb23841a45c78ff5cc7e492e0c14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:45.541852 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d38d75fcd494fc4a3420bcf15f2c70f consensus state has no leader
W20260501 13:57:45.554839 1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 71601.4us
I20260501 13:57:45.505098 2629 raft_consensus.cc:493] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 13:57:45.565722 1779 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.004s sys 0.000s
W20260501 13:57:45.566103 1779 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.559s user 0.006s sys 0.001s
I20260501 13:57:45.568132 1883 ts_tablet_manager.cc:1434] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.387s user 0.006s sys 0.000s
I20260501 13:57:45.586242 2633 tablet.cc:1724] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:45.591257 1881 raft_consensus.cc:359] T 406e1af8b7d9488f88697206daa87c05 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.591984 1881 raft_consensus.cc:385] T 406e1af8b7d9488f88697206daa87c05 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:45.592581 1881 raft_consensus.cc:740] T 406e1af8b7d9488f88697206daa87c05 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:45.593202 1881 consensus_queue.cc:260] T 406e1af8b7d9488f88697206daa87c05 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.606418 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d0fa1a12a554984a893ec75e8b32382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.606640 2456 raft_consensus.cc:493] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:45.607230 2456 raft_consensus.cc:515] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.610932 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a5f2c9b3e624fcebd2ddc86d66b08c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.617323 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaf868a5d2c24a45b183299d71f082a9" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:45.618901 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddeccfad2eb749bbbacf0a4543fff970. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:45.618809 1717 leader_election.cc:343] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.622891 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74af56cca54347bab67895e10ee17230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:45.623528 1883 tablet_bootstrap.cc:492] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:45.627430 2631 tablet.cc:1724] T ffe89d656956440e8d0c1de0b01ce486 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:45.573418 1132 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 1409fbe5846a4bb099f31fb2d2e41382 (table default.Table_be2275f0abfb43abb3bd0b1e128d7fef [id=8d39f1f13bc24a2c84e888bb29b6e74c]). Expected version 1 got 0
I20260501 13:57:45.635610 1132 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 406e1af8b7d9488f88697206daa87c05 (table default.Table_759f3009542147de94c73784e1698246 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]). Expected version 1 got 0
I20260501 13:57:45.636674 1132 catalog_manager.cc:5671] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:45.637856 1132 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:45.642565 1882 tablet_bootstrap.cc:654] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:45.652774 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_7bbaf820afc746198b26a791c87cb736" } modify_external_catalogs: true
I20260501 13:57:45.653306 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_7bbaf820afc746198b26a791c87cb736" } modify_external_catalogs: true
I20260501 13:57:45.609153 2456 leader_election.cc:290] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:45.654841 1124 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4a285dc47b4c471792aecb0dd81531ea" } modify_external_catalogs: true
I20260501 13:57:45.655334 1124 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4a285dc47b4c471792aecb0dd81531ea" } modify_external_catalogs: true
I20260501 13:57:45.662233 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_b92aef0dd8d843679f848b994f328edf" } new_table_name: "default.Table_34d3d9a3b69049028bd2692789a1a798" modify_external_catalogs: true
I20260501 13:57:45.667675 1675 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:45.675859 1132 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet ffe89d656956440e8d0c1de0b01ce486 (table default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715]). Expected version 1 got 0
I20260501 13:57:45.680487 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_cdc592203f384ccd8c0142007f14d5ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:45.682163 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdc592203f384ccd8c0142007f14d5ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:45.684820 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c20d4bb766404bb78ad87537eebc52c4 has no consensus state
I20260501 13:57:45.691514 1881 ts_tablet_manager.cc:1434] T 406e1af8b7d9488f88697206daa87c05 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.322s user 0.001s sys 0.005s
I20260501 13:57:45.692487 1881 tablet_bootstrap.cc:492] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:45.694856 1132 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 7f9c92846d4e4255a1c8ce3835a0a18d (table default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf]). Expected version 1 got 0
I20260501 13:57:45.695612 1132 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 96e937b3a0b5451e8eaf4d5c47c17620 (table default.Table_b29786a52abd4fad9a2337d7c76fbbc4 [id=05998ead56ed454dab9a71ef2ac9cb58]). Expected version 1 got 0
I20260501 13:57:45.699137 1124 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 994e61a306354f7b8475088d954c5b6c
I20260501 13:57:45.718004 2536 raft_consensus.cc:493] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:45.718494 2536 raft_consensus.cc:515] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.720281 2536 leader_election.cc:290] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:45.722355 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0356b19f0aa4197a41cc66d4440bffb" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:45.722898 1629 raft_consensus.cc:2468] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:45.723973 1421 leader_election.cc:304] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:45.724800 2536 raft_consensus.cc:2804] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:45.726393 2536 raft_consensus.cc:493] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:45.726735 2536 raft_consensus.cc:3060] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:45.721091 2047 ts_tablet_manager.cc:1929] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:45.732227 2047 log.cc:1199] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/39cd02738e5f4ecfbfa06d5541614ffd
I20260501 13:57:45.734649 2047 ts_tablet_manager.cc:1950] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:45.739696 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 39cd02738e5f4ecfbfa06d5541614ffd (table default.Table_6f9bd93befbf45c0a7415d32ce58d7fb [id=edb6ad6e0b444d3b8930c43ce4082a5c]) successfully deleted
I20260501 13:57:45.740487 2047 tablet_replica.cc:333] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:45.743655 2536 raft_consensus.cc:515] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.743655 2047 raft_consensus.cc:2243] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:45.747177 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 6399ac6a0d044ad2aed95ba30e6659b8
I20260501 13:57:45.747816 2047 raft_consensus.cc:2272] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:45.750685 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet 6399ac6a0d044ad2aed95ba30e6659b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:45.765406 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_19db51298b7949ebac6642911a90c5c7" } new_table_name: "default.Table_c81546b1614841bebe7a00f6ef1515dc" modify_external_catalogs: true
I20260501 13:57:45.770370 1131 master_service.cc:954] ReplaceTablet: received request to replace tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:45.752460 1462 tablet_service.cc:1558] Processing DeleteTablet for tablet 6399ac6a0d044ad2aed95ba30e6659b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:45.776180 2047 ts_tablet_manager.cc:1916] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:45.778144 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0356b19f0aa4197a41cc66d4440bffb" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:45.778689 1629 raft_consensus.cc:3060] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:45.787492 1629 raft_consensus.cc:2468] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:45.781781 2536 leader_election.cc:290] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:45.788789 1421 leader_election.cc:304] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters:
I20260501 13:57:45.753567 1881 tablet_bootstrap.cc:654] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:45.793128 1132 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_be2275f0abfb43abb3bd0b1e128d7fef [id=8d39f1f13bc24a2c84e888bb29b6e74c] alter complete (version 1)
I20260501 13:57:45.794303 2536 raft_consensus.cc:2804] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:45.794832 2536 raft_consensus.cc:697] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:45.795655 2536 consensus_queue.cc:237] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.804823 1127 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 406e1af8b7d9488f88697206daa87c05 (table default.Table_759f3009542147de94c73784e1698246 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]). Expected version 1 got 0
I20260501 13:57:45.808497 2623 raft_consensus.cc:493] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:45.808947 2623 raft_consensus.cc:515] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.809190 2564 ts_tablet_manager.cc:1929] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:45.810828 2623 leader_election.cc:290] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:45.814810 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "967658b1be204c638fc34983293ef5ab" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:45.815789 1719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 155435us
W20260501 13:57:45.816010 1717 leader_election.cc:343] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:45.816375 1719 leader_election.cc:343] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.816689 1719 leader_election.cc:304] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:45.817555 1719 leader_election.cc:343] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.817895 1719 leader_election.cc:304] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:45.823768 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet e1d7a80ad4c4455a9ddee7a955e32635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:45.824491 1478 raft_consensus.cc:3060] T 406e1af8b7d9488f88697206daa87c05 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:45.828328 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd02738e5f4ecfbfa06d5541614ffd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:45.830750 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet 67b0675398d04be08bd29638b4ba2083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:45.832791 2582 ts_tablet_manager.cc:1929] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:45.833163 2582 log.cc:1199] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/9abb39c8b44d4b4c8b69751a4cdd6668
I20260501 13:57:45.833776 2582 ts_tablet_manager.cc:1950] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:45.843607 1124 master_service.cc:954] ReplaceTablet: received request to replace tablet 30a1ef8502c648ce8da3905e4dff4c0c from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:45.843611 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_dc76e09f85ed44ff96e6f2f1a7a1c223"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:45.843847 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_62d1e55cf6304e208c648b8d70b41550"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:45.845271 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62d1e55cf6304e208c648b8d70b41550 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:45.846335 2047 ts_tablet_manager.cc:1929] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:45.849932 2047 log.cc:1199] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/a9672aab85c64d548a4f6583a1e9af20
W20260501 13:57:45.849967 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc76e09f85ed44ff96e6f2f1a7a1c223 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:45.850672 2047 ts_tablet_manager.cc:1950] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:45.851962 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet a9672aab85c64d548a4f6583a1e9af20 (table default.Table_bc1aee595536420799f8a28881253c09 [id=787286aa1fb44c45827992c1cf240e2d]) successfully deleted
I20260501 13:57:45.852815 2047 tablet_replica.cc:333] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
W20260501 13:57:45.860926 1126 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6df58010b9084f4ab4710dd8eb5c4041 has no consensus state
I20260501 13:57:45.864517 2047 raft_consensus.cc:2243] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:45.866082 2047 pending_rounds.cc:70] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: Trying to abort 1 pending ops.
I20260501 13:57:45.866365 2047 pending_rounds.cc:77] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229274484133888 op_type: NO_OP noop_request { }
I20260501 13:57:45.866787 2047 raft_consensus.cc:2889] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:45.867158 2047 raft_consensus.cc:2272] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:45.876401 1418 leader_election.cc:343] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 39cd02738e5f4ecfbfa06d5541614ffd
I20260501 13:57:45.882685 1127 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_6ca908dd49884d579ce8aa66535b611f [id=3c1a635628034d019c34ce2059bee715] alter complete (version 1)
I20260501 13:57:45.782198 2624 raft_consensus.cc:515] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:45.890914 1418 leader_election.cc:343] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: a9672aab85c64d548a4f6583a1e9af20
I20260501 13:57:45.891285 1418 leader_election.cc:304] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
W20260501 13:57:45.891777 1418 raft_consensus.cc:2686] T a9672aab85c64d548a4f6583a1e9af20 P 993b9ba18d8547f8af43167e727a6a24: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260501 13:57:45.891839 2624 leader_election.cc:290] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:45.894037 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c1e0c30dce488baed9e046d90f0ba4" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:45.830883 2564 log.cc:1199] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/68cb7bd1a5aa4419b9c5176a287fe885
I20260501 13:57:45.895872 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet d361a2d82ecd456fb43f5e7fe3c83094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:45.895802 1563 leader_election.cc:343] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: d361a2d82ecd456fb43f5e7fe3c83094
I20260501 13:57:45.896344 1563 leader_election.cc:304] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:45.897765 1418 leader_election.cc:343] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: e0356b19f0aa4197a41cc66d4440bffb
I20260501 13:57:45.900115 1883 tablet_bootstrap.cc:654] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:45.901180 2047 ts_tablet_manager.cc:1916] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:45.905429 1717 leader_election.cc:343] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.912346 2590 raft_consensus.cc:2749] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:45.912786 1881 tablet_bootstrap.cc:492] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:45.913168 1881 ts_tablet_manager.cc:1403] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.221s user 0.011s sys 0.010s
I20260501 13:57:45.916671 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ca91828e5c8642c3be033ef95e53f2a3" } new_table_name: "default.Table_d6d2733692544328a2329333c417d28d" modify_external_catalogs: true
I20260501 13:57:45.919981 1881 raft_consensus.cc:359] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.920756 1881 raft_consensus.cc:385] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:45.921049 1881 raft_consensus.cc:740] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:45.921777 1881 consensus_queue.cc:260] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.920419 1132 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_d23b951fef3048af97eb045b842ebd22" } modify_external_catalogs: true
I20260501 13:57:45.922798 1132 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_d23b951fef3048af97eb045b842ebd22" } modify_external_catalogs: true
I20260501 13:57:45.924417 1881 ts_tablet_manager.cc:1434] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260501 13:57:45.925266 1881 tablet_bootstrap.cc:492] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:45.925978 2590 consensus_queue.cc:1048] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:45.859014 2456 raft_consensus.cc:2749] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:45.859557 2623 raft_consensus.cc:2749] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:45.928184 1882 tablet_bootstrap.cc:492] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:45.928570 1882 ts_tablet_manager.cc:1403] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.431s user 0.010s sys 0.006s
I20260501 13:57:45.932119 1882 raft_consensus.cc:359] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.932763 1882 raft_consensus.cc:385] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:45.932535 2643 tablet.cc:1724] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:45.933007 1882 raft_consensus.cc:740] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:45.822449 2629 raft_consensus.cc:515] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.934010 2564 ts_tablet_manager.cc:1950] T 68cb7bd1a5aa4419b9c5176a287fe885 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:45.934908 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 29f0e9def65d4ba9ad5aded73ea20515 (DEFAULT_TABLE table=default.Table_ca91828e5c8642c3be033ef95e53f2a3 [id=b733e10eaf5a438ca21057e62556588a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:45.935623 2629 leader_election.cc:290] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:45.936262 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f0e9def65d4ba9ad5aded73ea20515. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:45.942243 1418 leader_election.cc:343] T 39cd02738e5f4ecfbfa06d5541614ffd P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 39cd02738e5f4ecfbfa06d5541614ffd
I20260501 13:57:45.943275 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_fdbac3c4239747e396088091988a1794"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:45.944837 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdbac3c4239747e396088091988a1794 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:45.945472 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_6d06a2f21f3140abb7a05eeee1b563ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:45.946877 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet eaf868a5d2c24a45b183299d71f082a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:45.947667 2564 tablet_replica.cc:333] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:45.947774 1755 tablet_service.cc:1558] Processing DeleteTablet for tablet 19c1e0c30dce488baed9e046d90f0ba4 with delete_type TABLET_DATA_DELETED (replaced by tablet ef191926320a4e0fa8bd94e47e9b9a86) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:45.948575 1753 tablet_service.cc:1558] Processing DeleteTablet for tablet 71ac54444a414e25bcd05452edfbf5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:45.948611 2564 raft_consensus.cc:2243] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:45.948844 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "967658b1be204c638fc34983293ef5ab" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:45.949215 2564 raft_consensus.cc:2272] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:45.949349 1760 tablet_service.cc:1558] Processing DeleteTablet for tablet 6399ac6a0d044ad2aed95ba30e6659b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:45.949994 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c1e0c30dce488baed9e046d90f0ba4" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:45.950433 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d06a2f21f3140abb7a05eeee1b563ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:45.951233 2564 ts_tablet_manager.cc:1916] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:45.951141 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4265d9ab3734a8b97d745dc02c6b845" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:45.953217 1757 tablet_service.cc:1558] Processing DeleteTablet for tablet ef622e34d46a4b19a59e916d22c2b18b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:45.953964 1418 leader_election.cc:343] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: e0356b19f0aa4197a41cc66d4440bffb
I20260501 13:57:45.953926 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaf868a5d2c24a45b183299d71f082a9" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:45.954886 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 68cb7bd1a5aa4419b9c5176a287fe885 (table default.Table_da800013eb694abeba00d698e4bbc439 [id=7ed508c1fff74966aa6f804d059b9c47]) successfully deleted
I20260501 13:57:45.955233 2590 raft_consensus.cc:493] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 13:57:45.955585 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 569a0240b1274d9bb830addbaf1ec343 because the tablet was not found. No further retry: Not found: Tablet not found: 569a0240b1274d9bb830addbaf1ec343
I20260501 13:57:45.955650 2590 raft_consensus.cc:515] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.956981 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4265d9ab3734a8b97d745dc02c6b845" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:45.957455 2590 leader_election.cc:290] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:45.958026 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94abe840b1994c0cb1f7ac7f4ac09cd9" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:45.958580 1717 leader_election.cc:343] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:45.959414 1564 leader_election.cc:343] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.959632 1128 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_511cd4142c9642d98428697731daef7a" } modify_external_catalogs: true
I20260501 13:57:45.960081 1128 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_511cd4142c9642d98428697731daef7a" } modify_external_catalogs: true
W20260501 13:57:45.961968 1719 leader_election.cc:343] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.962359 1719 leader_election.cc:304] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:45.963444 1719 leader_election.cc:343] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.966166 1719 leader_election.cc:304] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:45.969565 2456 raft_consensus.cc:2749] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:45.971909 1563 leader_election.cc:343] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 94abe840b1994c0cb1f7ac7f4ac09cd9
W20260501 13:57:45.973330 1719 leader_election.cc:343] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:45.973755 1563 leader_election.cc:304] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:45.979390 1564 consensus_peers.cc:597] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:45.980443 2520 raft_consensus.cc:2749] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:45.988396 1882 consensus_queue.cc:260] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:45.991827 1882 ts_tablet_manager.cc:1434] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.063s user 0.005s sys 0.000s
I20260501 13:57:45.994346 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 289) took 1476 ms. Trace:
I20260501 13:57:45.998898 2582 rpcz_store.cc:276] 0501 13:57:44.518134 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:44.571883 (+ 53749us) service_pool.cc:224] Handling call
0501 13:57:45.161354 (+589471us) ts_tablet_manager.cc:1151] Deleting tablet 9abb39c8b44d4b4c8b69751a4cdd6668
0501 13:57:45.161388 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:45.494207 (+332819us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:45.832670 (+338463us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:45.994269 (+161599us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7696,"tablet-delete.queue_time_us":588821}
I20260501 13:57:46.000406 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 9abb39c8b44d4b4c8b69751a4cdd6668 (table default.Table_2a23504aef0e48beaf3541c53fda054e [id=0e61d9e75f4c4b20a849a0540c83bd18]) successfully deleted
I20260501 13:57:46.000830 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 284) took 1484 ms. Trace:
I20260501 13:57:45.999747 1882 tablet_bootstrap.cc:492] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:45.973755 1719 leader_election.cc:304] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:46.003969 1719 leader_election.cc:343] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:46.006707 2456 raft_consensus.cc:2749] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:45.994695 2623 raft_consensus.cc:2749] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:46.011538 1719 leader_election.cc:304] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:46.015518 1881 tablet_bootstrap.cc:654] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:46.017483 1882 tablet_bootstrap.cc:654] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:46.019037 1125 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29f0e9def65d4ba9ad5aded73ea20515 consensus state has no leader
I20260501 13:57:46.031041 2590 raft_consensus.cc:493] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:46.031572 2590 raft_consensus.cc:515] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.033281 2590 leader_election.cc:290] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:46.034206 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51ba7b8a7f4d4ec6ba4af775596d3b59" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:46.035476 1564 leader_election.cc:343] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:46.043859 1881 tablet_bootstrap.cc:492] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:46.044286 1881 ts_tablet_manager.cc:1403] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.119s user 0.005s sys 0.008s
I20260501 13:57:46.061939 1882 tablet_bootstrap.cc:492] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:46.062384 1882 ts_tablet_manager.cc:1403] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.000s
W20260501 13:57:46.063309 1563 leader_election.cc:343] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 51ba7b8a7f4d4ec6ba4af775596d3b59
I20260501 13:57:46.063730 1563 leader_election.cc:304] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:46.001164 2582 rpcz_store.cc:276] 0501 13:57:44.516619 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:44.570966 (+ 54347us) service_pool.cc:224] Handling call
0501 13:57:46.000453 (+1429487us) ts_tablet_manager.cc:1151] Deleting tablet e8efee17e2c34caf98c0ede01d2a943a
0501 13:57:46.000500 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:46.000776 (+ 276us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1427320}
I20260501 13:57:46.064733 1882 raft_consensus.cc:359] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:46.065081 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet e8efee17e2c34caf98c0ede01d2a943a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8efee17e2c34caf98c0ede01d2a943a already in progress: creating tablet
I20260501 13:57:46.065347 1882 raft_consensus.cc:385] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:46.065670 1882 raft_consensus.cc:740] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:46.065943 2582 tablet_replica.cc:333] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:46.065295 2590 raft_consensus.cc:2749] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:46.068336 1130 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_bc6341b170b3425eb49b5cc95ed40eb2" } new_table_name: "default.Table_974bf2748588469c88a2d8d6f34b1c5e" modify_external_catalogs: true
I20260501 13:57:46.053877 1881 raft_consensus.cc:359] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.071851 1881 raft_consensus.cc:385] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:46.072154 1881 raft_consensus.cc:740] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:46.066231 1882 consensus_queue.cc:260] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.073105 1881 consensus_queue.cc:260] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:46.066816 1719 raft_consensus.cc:2686] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260501 13:57:46.074882 1719 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 70962.6us
I20260501 13:57:46.075409 2582 raft_consensus.cc:2243] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:46.075889 2582 raft_consensus.cc:2272] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:46.078114 2582 ts_tablet_manager.cc:1916] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:46.084882 1882 ts_tablet_manager.cc:1434] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260501 13:57:46.085844 1882 tablet_bootstrap.cc:492] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:46.099398 2623 raft_consensus.cc:493] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:46.099817 2623 raft_consensus.cc:515] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.101356 2564 ts_tablet_manager.cc:1929] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:46.101540 2623 leader_election.cc:290] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:46.101727 2564 log.cc:1199] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/eaf868a5d2c24a45b183299d71f082a9
I20260501 13:57:46.102519 2564 ts_tablet_manager.cc:1950] T eaf868a5d2c24a45b183299d71f082a9 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:46.103608 2564 tablet_replica.cc:333] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:46.106251 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d399de014d74cab8df14f44787ca2f7" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:46.106720 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_345347c7dbfc4eb4b62c842b166dbf0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:46.104220 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_96a8aa986c4a42bcbc6e2655bd886c5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.108239 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_345347c7dbfc4eb4b62c842b166dbf0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:46.108969 1717 leader_election.cc:343] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:46.109481 1130 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4f7d7867fbd4032869e9448f4993bbc consensus state has no leader
I20260501 13:57:46.111632 1881 ts_tablet_manager.cc:1434] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.067s user 0.005s sys 0.002s
I20260501 13:57:46.112465 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d399de014d74cab8df14f44787ca2f7" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:46.112771 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet eaf868a5d2c24a45b183299d71f082a9 (table default.Table_9f356943ea5a469c8eea733dc8231c36 [id=3780aea79eea4179b54f6cbd55f6d8f0]) successfully deleted
W20260501 13:57:46.108222 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96a8aa986c4a42bcbc6e2655bd886c5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:46.118722 1719 leader_election.cc:343] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:46.119082 1719 leader_election.cc:304] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:46.119997 2456 raft_consensus.cc:2749] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:46.121939 1881 tablet_bootstrap.cc:492] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:46.138228 2564 raft_consensus.cc:2243] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:46.138795 2564 raft_consensus.cc:2272] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:46.142344 1882 tablet_bootstrap.cc:654] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:46.143880 2564 ts_tablet_manager.cc:1916] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:46.155200 1881 tablet_bootstrap.cc:654] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:46.168109 1883 tablet_bootstrap.cc:492] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:46.168526 1883 ts_tablet_manager.cc:1403] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.545s user 0.023s sys 0.009s
I20260501 13:57:46.174850 1883 raft_consensus.cc:359] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.175493 1883 raft_consensus.cc:385] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:46.175779 1881 tablet_bootstrap.cc:492] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:46.175768 1883 raft_consensus.cc:740] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:46.176229 1881 ts_tablet_manager.cc:1403] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.054s user 0.012s sys 0.000s
I20260501 13:57:46.176604 2047 ts_tablet_manager.cc:1929] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:46.177270 1883 consensus_queue.cc:260] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.178097 2047 log.cc:1199] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/e0356b19f0aa4197a41cc66d4440bffb
I20260501 13:57:46.178830 2047 ts_tablet_manager.cc:1950] T e0356b19f0aa4197a41cc66d4440bffb P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:46.179491 1883 ts_tablet_manager.cc:1434] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260501 13:57:46.180384 1883 tablet_bootstrap.cc:492] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:46.185775 2643 tablet.cc:1724] T ffe89d656956440e8d0c1de0b01ce486 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:46.198741 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet e0356b19f0aa4197a41cc66d4440bffb (table default.Table_5be69bddaf9d4f6a8e87137a8a46e223 [id=d32e41f59f7d4b45842ea1b263eefbb6]) successfully deleted
I20260501 13:57:46.199998 1882 tablet_bootstrap.cc:492] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:46.200484 1882 ts_tablet_manager.cc:1403] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.115s user 0.012s sys 0.000s
I20260501 13:57:46.202785 1882 raft_consensus.cc:359] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.203475 1882 raft_consensus.cc:385] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:46.203788 1882 raft_consensus.cc:740] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:46.204396 1882 consensus_queue.cc:260] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.206377 1882 ts_tablet_manager.cc:1434] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260501 13:57:46.207768 1881 raft_consensus.cc:359] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.208428 1881 raft_consensus.cc:385] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:46.208706 1881 raft_consensus.cc:740] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:46.209362 1881 consensus_queue.cc:260] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:46.211434 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 71ac54444a414e25bcd05452edfbf5ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71ac54444a414e25bcd05452edfbf5ba already in progress: creating tablet
I20260501 13:57:46.213660 1881 ts_tablet_manager.cc:1434] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.037s user 0.004s sys 0.000s
W20260501 13:57:46.219147 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet ef622e34d46a4b19a59e916d22c2b18b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef622e34d46a4b19a59e916d22c2b18b already in progress: creating tablet
I20260501 13:57:46.223102 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_e2f41fc3135c458e9fda2d60189b4a23" } new_table_name: "default.Table_71bae277fb2a4b7cbd9308104bfea363" modify_external_catalogs: true
W20260501 13:57:46.229758 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 6399ac6a0d044ad2aed95ba30e6659b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6399ac6a0d044ad2aed95ba30e6659b8 already in progress: creating tablet
I20260501 13:57:46.232074 2564 ts_tablet_manager.cc:1929] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:46.232420 2564 log.cc:1199] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/19c1e0c30dce488baed9e046d90f0ba4
I20260501 13:57:46.233017 2564 ts_tablet_manager.cc:1950] T 19c1e0c30dce488baed9e046d90f0ba4 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:46.235035 1882 tablet_bootstrap.cc:492] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:46.247807 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_f7500aa3d01447d08cec470916e7407d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.249833 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ff55996178045c0981184330b903bb8 consensus state has no leader
W20260501 13:57:46.255694 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7500aa3d01447d08cec470916e7407d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:46.263278 1881 tablet_bootstrap.cc:492] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:46.265276 1883 tablet_bootstrap.cc:654] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:46.236176 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 19c1e0c30dce488baed9e046d90f0ba4 (table default.Table_66a4e496efa34deaa6c793a210a510ff [id=e8b19350d702499a82f41a6cce14dbd5]) successfully deleted
I20260501 13:57:46.266431 2623 raft_consensus.cc:493] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 13:57:46.266975 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 71ac54444a414e25bcd05452edfbf5ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71ac54444a414e25bcd05452edfbf5ba already in progress: creating tablet
I20260501 13:57:46.266849 2623 raft_consensus.cc:515] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:46.267553 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 6399ac6a0d044ad2aed95ba30e6659b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6399ac6a0d044ad2aed95ba30e6659b8 already in progress: creating tablet
W20260501 13:57:46.268076 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet ef622e34d46a4b19a59e916d22c2b18b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef622e34d46a4b19a59e916d22c2b18b already in progress: creating tablet
I20260501 13:57:46.268638 2623 leader_election.cc:290] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:46.270494 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f32170902c134c93a9aa71b2530c845e" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:46.272893 1717 leader_election.cc:343] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:46.272948 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f32170902c134c93a9aa71b2530c845e" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:46.273729 2644 raft_consensus.cc:493] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 13:57:46.274999 1719 leader_election.cc:343] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:46.275346 1719 leader_election.cc:304] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:46.274142 2644 raft_consensus.cc:515] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.277676 2644 leader_election.cc:290] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:46.277815 1882 tablet_bootstrap.cc:654] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:46.278353 2644 raft_consensus.cc:493] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:46.278760 2644 raft_consensus.cc:515] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:46.279711 1563 leader_election.cc:343] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 38c3eb388d474d718554775d65f58d78
I20260501 13:57:46.280589 2582 ts_tablet_manager.cc:1929] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:46.280941 2582 log.cc:1199] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/e0356b19f0aa4197a41cc66d4440bffb
I20260501 13:57:46.281553 2582 ts_tablet_manager.cc:1950] T e0356b19f0aa4197a41cc66d4440bffb P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:46.281555 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2ac522f2e4c4ab48a9b7d211c82ed9e" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:46.282814 1777 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2ac522f2e4c4ab48a9b7d211c82ed9e" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:46.283454 1777 raft_consensus.cc:2468] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:46.285987 2644 leader_election.cc:290] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:46.286847 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_39b2e9330e6f46b98c77bb74fbc11ac5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.288403 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39b2e9330e6f46b98c77bb74fbc11ac5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:46.289172 1564 leader_election.cc:343] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:46.289904 1563 leader_election.cc:304] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:46.290237 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38c3eb388d474d718554775d65f58d78" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:46.290987 2644 raft_consensus.cc:2804] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:46.291292 1881 tablet_bootstrap.cc:654] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:46.291357 2644 raft_consensus.cc:493] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260501 13:57:46.291584 1564 leader_election.cc:343] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:46.291746 2644 raft_consensus.cc:3060] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:46.292182 2623 raft_consensus.cc:2749] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:46.291924 1564 leader_election.cc:304] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:46.293321 2590 raft_consensus.cc:2749] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:46.302304 1532 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:46.282274 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 291) took 1752 ms. Trace:
I20260501 13:57:46.302747 2582 rpcz_store.cc:276] 0501 13:57:44.530097 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:44.580256 (+ 50159us) service_pool.cc:224] Handling call
0501 13:57:46.065782 (+1485526us) ts_tablet_manager.cc:1151] Deleting tablet e0356b19f0aa4197a41cc66d4440bffb
0501 13:57:46.065826 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:46.226906 (+161080us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:46.280473 (+ 53567us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:46.282214 (+ 1741us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1484739}
I20260501 13:57:46.303958 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet e0356b19f0aa4197a41cc66d4440bffb (table default.Table_5be69bddaf9d4f6a8e87137a8a46e223 [id=d32e41f59f7d4b45842ea1b263eefbb6]) successfully deleted
I20260501 13:57:46.312647 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_bc1658a0432141c1b6e439a4e36ffc33" } new_table_name: "default.Table_d564e785b85540ce834d43b55fda7ff0" modify_external_catalogs: true
I20260501 13:57:46.328739 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_fe1f71fadf9546cd98a2f81acc82bce6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.330243 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe1f71fadf9546cd98a2f81acc82bce6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:46.333357 2582 tablet_replica.cc:333] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:46.334237 2582 raft_consensus.cc:2243] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:46.334821 2582 raft_consensus.cc:2272] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:46.352137 1882 tablet_bootstrap.cc:492] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:46.352602 1882 ts_tablet_manager.cc:1403] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.118s user 0.010s sys 0.002s
I20260501 13:57:46.361713 1881 tablet_bootstrap.cc:492] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:46.362185 1881 ts_tablet_manager.cc:1403] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.099s user 0.010s sys 0.002s
W20260501 13:57:46.363710 1125 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 678de81f2adb4ae6a2fce7fbfd1b08e7 has no consensus state
I20260501 13:57:46.364462 1881 raft_consensus.cc:359] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.365034 1881 raft_consensus.cc:385] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:46.365303 1881 raft_consensus.cc:740] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:46.365869 1881 consensus_queue.cc:260] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.366245 2582 ts_tablet_manager.cc:1916] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:46.368477 1882 raft_consensus.cc:359] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.369516 1882 raft_consensus.cc:385] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:46.370021 1882 raft_consensus.cc:740] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
W20260501 13:57:46.373023 2614 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 9156742303377354975) tablet 00000000000000000000000000000000 cur holder 0x7b5800031500
I20260501 13:57:46.372308 1882 consensus_queue.cc:260] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.374986 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_3d46da834ff64e5b9b54a0a28295ed4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.376595 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d46da834ff64e5b9b54a0a28295ed4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:46.394734 1882 ts_tablet_manager.cc:1434] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.042s user 0.004s sys 0.001s
I20260501 13:57:46.395131 2554 raft_consensus.cc:493] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:46.395552 2554 raft_consensus.cc:515] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.397864 2554 leader_election.cc:290] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:46.399823 1125 master_service.cc:954] ReplaceTablet: received request to replace tablet a000b129a6ab429889b543c20750a493 from {username='slave'} at 127.0.0.1:41700
W20260501 13:57:46.407490 1418 leader_election.cc:343] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: ec4bd4cb1e70472aa09fc93afc3b1501
I20260501 13:57:46.407748 1126 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 967658b1be204c638fc34983293ef5ab (table default.Table_1ec94dd4afbb40d19a6f68aaefad8790 [id=f57f3fcda9444fa0a1ad062576f8d051]). Expected version 3 got 0
I20260501 13:57:46.413407 1127 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_b29786a52abd4fad9a2337d7c76fbbc4 [id=05998ead56ed454dab9a71ef2ac9cb58] alter complete (version 1)
I20260501 13:57:46.435932 1881 ts_tablet_manager.cc:1434] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.073s user 0.004s sys 0.000s
I20260501 13:57:46.402606 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec4bd4cb1e70472aa09fc93afc3b1501" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:46.439631 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 52bf98fd1f3542a0aeba8d3e8262aa25 (DEFAULT_TABLE table=default.Table_151d21fe56664c948925ce791cd0a061 [id=8646f8f617634f8eb31a72c1946c7096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.421604 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 with delete_type TABLET_DATA_DELETED (replaced by tablet f28db04617a449af8d87a554fb63e1e4) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.421604 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.440675 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52bf98fd1f3542a0aeba8d3e8262aa25. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:46.441004 1116 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 678de81f2adb4ae6a2fce7fbfd1b08e7 (table default.Table_d564e785b85540ce834d43b55fda7ff0 [id=bbb98b9cca114562b0cefa20c58ac794], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 678de81f2adb4ae6a2fce7fbfd1b08e7 has no consensus state
I20260501 13:57:46.442180 2644 raft_consensus.cc:515] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.448698 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 52bf98fd1f3542a0aeba8d3e8262aa25 (DEFAULT_TABLE table=default.Table_151d21fe56664c948925ce791cd0a061 [id=8646f8f617634f8eb31a72c1946c7096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.449684 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52bf98fd1f3542a0aeba8d3e8262aa25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.450944 1881 tablet_bootstrap.cc:492] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:46.453598 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_7a4f0da973284ac49c7dd7ff3e99e09f" } new_table_name: "default.Table_9fec3571b60d4a7da2653ad41120b531" modify_external_catalogs: true
I20260501 13:57:46.445971 1882 tablet_bootstrap.cc:492] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:46.460234 1126 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:46.467522 1883 tablet_bootstrap.cc:492] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:46.467945 1883 ts_tablet_manager.cc:1403] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.288s user 0.016s sys 0.004s
I20260501 13:57:46.472818 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet f32170902c134c93a9aa71b2530c845e with delete_type TABLET_DATA_DELETED (replaced by tablet 3b0ed592fd3642c194bb946c8c6003b4) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:46.473487 2564 tablet_replica.cc:333] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:46.476454 1601 tablet_service.cc:1511] Processing CreateTablet for tablet d5b35ca7cf634691a0eda1f197204903 (DEFAULT_TABLE table=default.Table_d547fe7eda17499384b316b482c599f0 [id=580b81a4a81d41cba8e3f8e46c72af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.477581 2564 raft_consensus.cc:2243] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:46.477994 2564 raft_consensus.cc:2272] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:46.479450 1883 raft_consensus.cc:359] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.480046 1883 raft_consensus.cc:385] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:46.480274 1883 raft_consensus.cc:740] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:46.480772 2564 ts_tablet_manager.cc:1916] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:46.480880 1883 consensus_queue.cc:260] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.483183 1132 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd
I20260501 13:57:46.483348 1883 ts_tablet_manager.cc:1434] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.015s user 0.004s sys 0.001s
I20260501 13:57:46.484151 1883 tablet_bootstrap.cc:492] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:46.486189 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 30a1ef8502c648ce8da3905e4dff4c0c
I20260501 13:57:46.487021 1128 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 6c7e65dad084440787754e24cdd579d9
I20260501 13:57:46.487597 1124 catalog_manager.cc:6741] ReplaceTablet: tablet 30a1ef8502c648ce8da3905e4dff4c0c deleted and replaced by tablet e7db988380fe4860881532072b10eefc
I20260501 13:57:46.488415 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 30a1ef8502c648ce8da3905e4dff4c0c with delete_type TABLET_DATA_DELETED (replaced by tablet e7db988380fe4860881532072b10eefc) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:46.489553 1761 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c7e65dad084440787754e24cdd579d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:46.490120 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.490338 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet 30a1ef8502c648ce8da3905e4dff4c0c with delete_type TABLET_DATA_DELETED (replaced by tablet e7db988380fe4860881532072b10eefc) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.491369 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c7e65dad084440787754e24cdd579d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.494748 1129 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 406e1af8b7d9488f88697206daa87c05 (table default.Table_759f3009542147de94c73784e1698246 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]). Expected version 1 got 0
I20260501 13:57:46.499701 1779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2ac522f2e4c4ab48a9b7d211c82ed9e" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:46.500267 1779 raft_consensus.cc:3060] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:46.503006 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2ac522f2e4c4ab48a9b7d211c82ed9e" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
W20260501 13:57:46.504117 1421 leader_election.cc:343] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: Tablet not RUNNING: STOPPED
I20260501 13:57:46.504504 1421 leader_election.cc:304] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
W20260501 13:57:46.505576 1564 leader_election.cc:343] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:46.507560 2657 raft_consensus.cc:2749] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:46.508872 2554 raft_consensus.cc:493] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:46.509330 2554 raft_consensus.cc:515] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.510406 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5b35ca7cf634691a0eda1f197204903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.511421 2554 leader_election.cc:290] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:46.512468 1418 leader_election.cc:343] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 0669a229d6d54eacac9889ad18e06f30
I20260501 13:57:46.513885 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 52bf98fd1f3542a0aeba8d3e8262aa25 (DEFAULT_TABLE table=default.Table_151d21fe56664c948925ce791cd0a061 [id=8646f8f617634f8eb31a72c1946c7096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.513506 1761 tablet_service.cc:1511] Processing CreateTablet for tablet d5b35ca7cf634691a0eda1f197204903 (DEFAULT_TABLE table=default.Table_d547fe7eda17499384b316b482c599f0 [id=580b81a4a81d41cba8e3f8e46c72af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.514168 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet 30a1ef8502c648ce8da3905e4dff4c0c with delete_type TABLET_DATA_DELETED (replaced by tablet e7db988380fe4860881532072b10eefc) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:46.514853 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52bf98fd1f3542a0aeba8d3e8262aa25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.515699 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c7e65dad084440787754e24cdd579d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:46.515574 1779 raft_consensus.cc:2468] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
W20260501 13:57:46.516932 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 30a1ef8502c648ce8da3905e4dff4c0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30a1ef8502c648ce8da3905e4dff4c0c already in progress: creating tablet
I20260501 13:57:46.516999 1563 leader_election.cc:304] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:46.517714 2520 raft_consensus.cc:2804] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
W20260501 13:57:46.517885 1117 catalog_manager.cc:4969] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 6c7e65dad084440787754e24cdd579d9 because the tablet was not found. No further retry: Not found: Tablet not found: 6c7e65dad084440787754e24cdd579d9
I20260501 13:57:46.518119 2520 raft_consensus.cc:697] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:46.519053 1456 tablet_service.cc:1511] Processing CreateTablet for tablet d5b35ca7cf634691a0eda1f197204903 (DEFAULT_TABLE table=default.Table_d547fe7eda17499384b316b482c599f0 [id=580b81a4a81d41cba8e3f8e46c72af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.518849 2520 consensus_queue.cc:237] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.519933 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5b35ca7cf634691a0eda1f197204903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.522670 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:46.523905 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0669a229d6d54eacac9889ad18e06f30" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:46.524427 1629 raft_consensus.cc:2468] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
I20260501 13:57:46.530720 2644 leader_election.cc:290] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:46.535184 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd already in progress: creating tablet
I20260501 13:57:46.536486 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet 9abb39c8b44d4b4c8b69751a4cdd6668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:32 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:46.536859 2582 ts_tablet_manager.cc:1929] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:46.537123 2047 tablet_replica.cc:333] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:46.537254 2582 log.cc:1199] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/ec4bd4cb1e70472aa09fc93afc3b1501
I20260501 13:57:46.537823 2047 raft_consensus.cc:2243] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:46.537868 2582 ts_tablet_manager.cc:1950] T ec4bd4cb1e70472aa09fc93afc3b1501 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:46.538256 2047 raft_consensus.cc:2272] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:46.540210 2047 ts_tablet_manager.cc:1916] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:46.542560 2657 raft_consensus.cc:493] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:46.543783 2657 raft_consensus.cc:515] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.544243 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5b35ca7cf634691a0eda1f197204903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.551369 2657 leader_election.cc:290] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:46.554828 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet 67b0675398d04be08bd29638b4ba2083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:46.555193 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet 0669a229d6d54eacac9889ad18e06f30 with delete_type TABLET_DATA_DELETED (replaced by tablet 81230beeff8d434aa2dac467c35d9703) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:46.556166 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet ec4bd4cb1e70472aa09fc93afc3b1501 with delete_type TABLET_DATA_DELETED (replaced by tablet 8dbb619863dd405ab20ead6bbebf4d69) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:46.556387 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67b0675398d04be08bd29638b4ba2083" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:46.556944 1762 tablet_service.cc:1558] Processing DeleteTablet for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:46.556954 1629 raft_consensus.cc:2468] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
W20260501 13:57:46.558234 1418 leader_election.cc:343] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 67b0675398d04be08bd29638b4ba2083
I20260501 13:57:46.566710 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_5bbee1238b46474e808a9fca817eefa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.568198 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bbee1238b46474e808a9fca817eefa0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:46.570923 1129 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_5d773086180e4cd09927c8b2fe2e650b" } new_table_name: "default.Table_d4ff0a7de20748ee910662913831001c" modify_external_catalogs: true
I20260501 13:57:46.575001 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_ce82bd9de8db4f84bb12d673fc45e30f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.576411 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce82bd9de8db4f84bb12d673fc45e30f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:46.580751 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 292) took 1537 ms. Trace:
I20260501 13:57:46.581130 2582 rpcz_store.cc:276] 0501 13:57:45.043264 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.043742 (+ 478us) service_pool.cc:224] Handling call
0501 13:57:46.310753 (+1267011us) ts_tablet_manager.cc:1151] Deleting tablet ec4bd4cb1e70472aa09fc93afc3b1501
0501 13:57:46.333203 (+ 22450us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:46.395776 (+ 62573us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:46.536737 (+140961us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:46.580686 (+ 43949us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266463}
I20260501 13:57:46.582262 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet ec4bd4cb1e70472aa09fc93afc3b1501 (table default.Table_6ee51a9fc8ae4340b14c506cf9c450c2 [id=e61fe8a70f66497db17db0afacc35100]) successfully deleted
I20260501 13:57:46.585350 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:46.591936 1421 leader_election.cc:304] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:46.592630 2657 raft_consensus.cc:2804] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:46.592969 2657 raft_consensus.cc:493] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:46.593267 2657 raft_consensus.cc:3060] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:46.597311 2564 ts_tablet_manager.cc:1929] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:46.597689 2564 log.cc:1199] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/f32170902c134c93a9aa71b2530c845e
I20260501 13:57:46.598304 2564 ts_tablet_manager.cc:1950] T f32170902c134c93a9aa71b2530c845e P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:46.593246 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 294) took 1549 ms. Trace:
I20260501 13:57:46.598642 2582 rpcz_store.cc:276] 0501 13:57:45.043842 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.043939 (+ 97us) service_pool.cc:224] Handling call
0501 13:57:46.592839 (+1548900us) ts_tablet_manager.cc:1151] Deleting tablet 53d9a8503c8d4bebb89ea493b99bf6fd
0501 13:57:46.592886 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:46.593189 (+ 303us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548269}
I20260501 13:57:46.599012 1421 leader_election.cc:304] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
W20260501 13:57:46.599751 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 53d9a8503c8d4bebb89ea493b99bf6fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53d9a8503c8d4bebb89ea493b99bf6fd already in progress: creating tablet
I20260501 13:57:46.599792 2658 raft_consensus.cc:2804] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:46.600100 2658 raft_consensus.cc:493] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:46.600380 2658 raft_consensus.cc:3060] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:46.604012 1882 tablet_bootstrap.cc:654] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:46.607940 1455 tablet_service.cc:1511] Processing CreateTablet for tablet 2aca712596e440b797010df853137b16 (DEFAULT_TABLE table=default.Table_524cc370fa15424ca5fb7c094c345897 [id=5a3920437c8e414483bde78d874173de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.610360 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 2aca712596e440b797010df853137b16 (DEFAULT_TABLE table=default.Table_524cc370fa15424ca5fb7c094c345897 [id=5a3920437c8e414483bde78d874173de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.611982 1465 tablet_service.cc:1511] Processing CreateTablet for tablet d38f1d9400934042b4b23bbb5b4985a5 (DEFAULT_TABLE table=default.Table_00e5296f8b75468ba13600c8afa47ce7 [id=4e2bfd498e0c475bade7797ad0223717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.614850 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_b62edaca3f654cd6a4dcd07badd05637"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:46.615974 1601 tablet_service.cc:1511] Processing CreateTablet for tablet 2aca712596e440b797010df853137b16 (DEFAULT_TABLE table=default.Table_524cc370fa15424ca5fb7c094c345897 [id=5a3920437c8e414483bde78d874173de]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:46.616272 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b62edaca3f654cd6a4dcd07badd05637 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:46.615165 1125 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet a000b129a6ab429889b543c20750a493
I20260501 13:57:46.618587 1125 catalog_manager.cc:6741] ReplaceTablet: tablet a000b129a6ab429889b543c20750a493 deleted and replaced by tablet 36b9a9c6e5c340f68b071c9a9ae8d464
I20260501 13:57:46.618841 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet a000b129a6ab429889b543c20750a493 with delete_type TABLET_DATA_DELETED (replaced by tablet 36b9a9c6e5c340f68b071c9a9ae8d464) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.613876 2658 raft_consensus.cc:515] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.620157 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet a000b129a6ab429889b543c20750a493 with delete_type TABLET_DATA_DELETED (replaced by tablet 36b9a9c6e5c340f68b071c9a9ae8d464) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:46.620271 2657 raft_consensus.cc:515] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.620473 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aca712596e440b797010df853137b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.621889 2658 leader_election.cc:290] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:46.626191 1418 leader_election.cc:343] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 67b0675398d04be08bd29638b4ba2083
I20260501 13:57:46.627518 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67b0675398d04be08bd29638b4ba2083" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:46.627979 1629 raft_consensus.cc:3060] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:46.629626 2047 ts_tablet_manager.cc:1929] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:46.629936 2047 log.cc:1199] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/9abb39c8b44d4b4c8b69751a4cdd6668
I20260501 13:57:46.630514 2047 ts_tablet_manager.cc:1950] T 9abb39c8b44d4b4c8b69751a4cdd6668 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:46.631366 2582 tablet_replica.cc:333] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
W20260501 13:57:46.632803 1129 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea425f73649546648a1082ee75b8f116 consensus state has no leader
W20260501 13:57:46.632979 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 301fad71fe6e4e53a36216c41f2850bf consensus state has no leader
I20260501 13:57:46.632174 2582 raft_consensus.cc:2243] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:46.614537 1881 tablet_bootstrap.cc:654] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:46.617981 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet a000b129a6ab429889b543c20750a493 with delete_type TABLET_DATA_DELETED (replaced by tablet 36b9a9c6e5c340f68b071c9a9ae8d464) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:46.637820 2582 raft_consensus.cc:2272] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:46.638008 1629 raft_consensus.cc:2468] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:46.639189 1421 leader_election.cc:304] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:46.641817 2658 raft_consensus.cc:2804] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:46.642695 2658 raft_consensus.cc:697] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:46.643541 2658 consensus_queue.cc:237] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.647153 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aca712596e440b797010df853137b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.649538 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet f32170902c134c93a9aa71b2530c845e (table default.Table_407a6cfd065142f4a2debee031077ab9 [id=a5184058a20f431280f6ccd10a290f27]) successfully deleted
W20260501 13:57:46.650916 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 30a1ef8502c648ce8da3905e4dff4c0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30a1ef8502c648ce8da3905e4dff4c0c already in progress: creating tablet
W20260501 13:57:46.651547 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 6c7e65dad084440787754e24cdd579d9 because the tablet was not found. No further retry: Not found: Tablet not found: 6c7e65dad084440787754e24cdd579d9
W20260501 13:57:46.652105 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd already in progress: creating tablet
I20260501 13:57:46.658011 1756 tablet_service.cc:1511] Processing CreateTablet for tablet d38f1d9400934042b4b23bbb5b4985a5 (DEFAULT_TABLE table=default.Table_00e5296f8b75468ba13600c8afa47ce7 [id=4e2bfd498e0c475bade7797ad0223717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.659909 1609 tablet_service.cc:1511] Processing CreateTablet for tablet d38f1d9400934042b4b23bbb5b4985a5 (DEFAULT_TABLE table=default.Table_00e5296f8b75468ba13600c8afa47ce7 [id=4e2bfd498e0c475bade7797ad0223717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.660328 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38f1d9400934042b4b23bbb5b4985a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.660826 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38f1d9400934042b4b23bbb5b4985a5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:46.666706 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet a000b129a6ab429889b543c20750a493 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a000b129a6ab429889b543c20750a493 already in progress: creating tablet
I20260501 13:57:46.666949 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 665a8d647e494b44b17ad6cdb02337c8 (DEFAULT_TABLE table=default.Table_86cb74d55d764d4abd4f2542b889ecf2 [id=d8ba2ed44365478da3a569fb9cec1121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.673216 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c736959df334aee857dbaae0f7a1e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:40 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.674614 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet 69626ed095bd4df7859563c176dafffc with delete_type TABLET_DATA_DELETED (replaced by tablet 1042bdf58c1f488daeaec284eb3319d5) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.675961 2582 ts_tablet_manager.cc:1916] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:46.677029 1882 tablet_bootstrap.cc:492] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:46.677723 1882 ts_tablet_manager.cc:1403] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.232s user 0.010s sys 0.000s
I20260501 13:57:46.677951 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_aac3e9446b9e433a9cd6c8a08f7b7660"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.679399 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aac3e9446b9e433a9cd6c8a08f7b7660 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:46.679718 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aca712596e440b797010df853137b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.680238 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38f1d9400934042b4b23bbb5b4985a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.683467 2657 leader_election.cc:290] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:46.684690 1418 leader_election.cc:343] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 0669a229d6d54eacac9889ad18e06f30
I20260501 13:57:46.686709 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0669a229d6d54eacac9889ad18e06f30" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:46.688144 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 9abb39c8b44d4b4c8b69751a4cdd6668 (table default.Table_2a23504aef0e48beaf3541c53fda054e [id=0e61d9e75f4c4b20a849a0540c83bd18]) successfully deleted
W20260501 13:57:46.690352 1421 leader_election.cc:343] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: Tablet not RUNNING: STOPPED
I20260501 13:57:46.690769 2047 tablet_replica.cc:333] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:46.697796 1124 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_fe9befde21164496ad12d593c8e08660" } modify_external_catalogs: true
I20260501 13:57:46.698300 1124 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_fe9befde21164496ad12d593c8e08660" } modify_external_catalogs: true
I20260501 13:57:46.700186 1132 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_bd8182bfb41b4025b826d479090f8f09" } modify_external_catalogs: true
I20260501 13:57:46.700661 1132 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_bd8182bfb41b4025b826d479090f8f09" } modify_external_catalogs: true
I20260501 13:57:46.701586 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 665a8d647e494b44b17ad6cdb02337c8 (DEFAULT_TABLE table=default.Table_86cb74d55d764d4abd4f2542b889ecf2 [id=d8ba2ed44365478da3a569fb9cec1121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.702545 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 665a8d647e494b44b17ad6cdb02337c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.711993 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 665a8d647e494b44b17ad6cdb02337c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.712360 2644 raft_consensus.cc:493] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:46.712853 2644 raft_consensus.cc:515] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.715889 2644 leader_election.cc:290] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:46.717187 1563 leader_election.cc:343] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 0c736959df334aee857dbaae0f7a1e1a
I20260501 13:57:46.725324 1462 tablet_service.cc:1511] Processing CreateTablet for tablet 6df58010b9084f4ab4710dd8eb5c4041 (DEFAULT_TABLE table=default.Table_c81546b1614841bebe7a00f6ef1515dc [id=9f6404af5211431fa5f99fb7abeea51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.728972 1882 raft_consensus.cc:359] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.730299 1882 raft_consensus.cc:385] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:46.726265 1754 tablet_service.cc:1511] Processing CreateTablet for tablet 665a8d647e494b44b17ad6cdb02337c8 (DEFAULT_TABLE table=default.Table_86cb74d55d764d4abd4f2542b889ecf2 [id=d8ba2ed44365478da3a569fb9cec1121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.730520 2047 raft_consensus.cc:2243] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:46.730821 1882 raft_consensus.cc:740] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:46.731331 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 665a8d647e494b44b17ad6cdb02337c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.731704 2047 pending_rounds.cc:70] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: Trying to abort 1 pending ops.
I20260501 13:57:46.731918 2047 pending_rounds.cc:77] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229277782261760 op_type: NO_OP noop_request { }
I20260501 13:57:46.732236 2047 raft_consensus.cc:2889] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:46.732530 2047 raft_consensus.cc:2272] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:46.731896 1882 consensus_queue.cc:260] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.690804 1421 leader_election.cc:304] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:46.739246 2658 raft_consensus.cc:2749] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260501 13:57:46.739357 1421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 49044.9us
I20260501 13:57:46.740301 1609 tablet_service.cc:1558] Processing DeleteTablet for tablet 38c3eb388d474d718554775d65f58d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:46.746125 2665 log.cc:927] Time spent T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260501 13:57:46.747150 2665 consensus_queue.cc:1175] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "993b9ba18d8547f8af43167e727a6a24" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20260501 13:57:46.747398 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 6df58010b9084f4ab4710dd8eb5c4041 (DEFAULT_TABLE table=default.Table_c81546b1614841bebe7a00f6ef1515dc [id=9f6404af5211431fa5f99fb7abeea51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.748310 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6df58010b9084f4ab4710dd8eb5c4041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.749002 1883 tablet_bootstrap.cc:654] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:46.754076 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet bd633c026195429ca69d8041809bee45
I20260501 13:57:46.756021 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet bd633c026195429ca69d8041809bee45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:46 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:46.756832 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet bd633c026195429ca69d8041809bee45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:46 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.757122 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet bd633c026195429ca69d8041809bee45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:46 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:46.759148 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet bd633c026195429ca69d8041809bee45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd633c026195429ca69d8041809bee45 already in progress: creating tablet
I20260501 13:57:46.760421 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c736959df334aee857dbaae0f7a1e1a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:46.763883 1881 tablet_bootstrap.cc:492] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:46.764340 1881 ts_tablet_manager.cc:1403] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.314s user 0.010s sys 0.000s
I20260501 13:57:46.766587 1881 raft_consensus.cc:359] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:46.770371 1564 leader_election.cc:343] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:46.770787 1564 leader_election.cc:304] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:46.771664 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6df58010b9084f4ab4710dd8eb5c4041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.773533 1881 raft_consensus.cc:385] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:46.773823 1881 raft_consensus.cc:740] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:46.774410 1881 consensus_queue.cc:260] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.777815 1762 tablet_service.cc:1511] Processing CreateTablet for tablet 6df58010b9084f4ab4710dd8eb5c4041 (DEFAULT_TABLE table=default.Table_c81546b1614841bebe7a00f6ef1515dc [id=9f6404af5211431fa5f99fb7abeea51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.784528 1132 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 37f57009c05f479481b59e49062608d5
I20260501 13:57:46.786541 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet 37f57009c05f479481b59e49062608d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:46 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:46.787604 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6df58010b9084f4ab4710dd8eb5c4041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.791492 1881 ts_tablet_manager.cc:1434] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20260501 13:57:46.795076 2590 raft_consensus.cc:2749] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:46.796036 1610 tablet_service.cc:1558] Processing DeleteTablet for tablet 37f57009c05f479481b59e49062608d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:46 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.800020 1881 tablet_bootstrap.cc:492] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:46.800861 1465 tablet_service.cc:1511] Processing CreateTablet for tablet bbb10f2c3d864931abb3ae2a88372d61 (DEFAULT_TABLE table=default.Table_ca507f190ca9413d91939e27911c79d1 [id=455cd3835ef44ef3a217afc8eb8ee3d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.802122 1610 tablet_service.cc:1511] Processing CreateTablet for tablet bbb10f2c3d864931abb3ae2a88372d61 (DEFAULT_TABLE table=default.Table_ca507f190ca9413d91939e27911c79d1 [id=455cd3835ef44ef3a217afc8eb8ee3d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.803750 1756 tablet_service.cc:1511] Processing CreateTablet for tablet bbb10f2c3d864931abb3ae2a88372d61 (DEFAULT_TABLE table=default.Table_ca507f190ca9413d91939e27911c79d1 [id=455cd3835ef44ef3a217afc8eb8ee3d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.804792 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbb10f2c3d864931abb3ae2a88372d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.783118 2047 ts_tablet_manager.cc:1916] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:46.789311 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet 37f57009c05f479481b59e49062608d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:46 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:46.807785 1882 ts_tablet_manager.cc:1434] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.130s user 0.006s sys 0.000s
I20260501 13:57:46.813129 1883 tablet_bootstrap.cc:492] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:46.813536 1883 ts_tablet_manager.cc:1403] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.330s user 0.005s sys 0.004s
I20260501 13:57:46.812896 1126 catalog_manager.cc:5671] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:46.825539 1455 tablet_service.cc:1511] Processing CreateTablet for tablet 05cfbdc8bc174f9698a5aa9a8e30ca04 (DEFAULT_TABLE table=default.Table_b1d9b08159ee48c7bb3e453da9f6ff40 [id=4e7e8da979a44dc78e657f586a602732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.827941 1602 tablet_service.cc:1511] Processing CreateTablet for tablet 05cfbdc8bc174f9698a5aa9a8e30ca04 (DEFAULT_TABLE table=default.Table_b1d9b08159ee48c7bb3e453da9f6ff40 [id=4e7e8da979a44dc78e657f586a602732]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:46.831952 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 37f57009c05f479481b59e49062608d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37f57009c05f479481b59e49062608d5 already in progress: creating tablet
I20260501 13:57:46.833973 1881 tablet_bootstrap.cc:654] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:46.839064 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_43f068125a9243dbadd21b551da3226b" } new_table_name: "default.Table_74984006d27c461b927ebb211fffc155" modify_external_catalogs: true
I20260501 13:57:46.839356 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_345347c7dbfc4eb4b62c842b166dbf0f" } new_table_name: "default.Table_7985a0b2ef3a44e6ab6fc2a8d062198f" modify_external_catalogs: true
I20260501 13:57:46.841288 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4e3867de9f9848d99ecb5f0dd886d117"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.842721 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e3867de9f9848d99ecb5f0dd886d117 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:46.843729 1883 raft_consensus.cc:359] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.844339 1883 raft_consensus.cc:385] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:46.844579 1883 raft_consensus.cc:740] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:46.847110 1761 tablet_service.cc:1511] Processing CreateTablet for tablet 05cfbdc8bc174f9698a5aa9a8e30ca04 (DEFAULT_TABLE table=default.Table_b1d9b08159ee48c7bb3e453da9f6ff40 [id=4e7e8da979a44dc78e657f586a602732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.854362 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_ea2a86f75e984168b2aeb1606559389d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.855909 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea2a86f75e984168b2aeb1606559389d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:46.845127 1883 consensus_queue.cc:260] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.864979 1608 tablet_service.cc:1511] Processing CreateTablet for tablet 9b2e6821c81b4eb2bf8cc283db989515 (DEFAULT_TABLE table=default.Table_295132c6928c4b309137696f9015de9f [id=2b17c3fbec3747309803bc86cbe42d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.866544 1754 tablet_service.cc:1511] Processing CreateTablet for tablet 9b2e6821c81b4eb2bf8cc283db989515 (DEFAULT_TABLE table=default.Table_295132c6928c4b309137696f9015de9f [id=2b17c3fbec3747309803bc86cbe42d5c]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:46.869644 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96875c3b0f7f4421ab5d9f65ba4260a2 consensus state has no leader
I20260501 13:57:46.877178 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbb10f2c3d864931abb3ae2a88372d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.877391 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet d361a2d82ecd456fb43f5e7fe3c83094 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:36 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:46.877868 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05cfbdc8bc174f9698a5aa9a8e30ca04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.886797 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05cfbdc8bc174f9698a5aa9a8e30ca04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.891361 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 9b2e6821c81b4eb2bf8cc283db989515 (DEFAULT_TABLE table=default.Table_295132c6928c4b309137696f9015de9f [id=2b17c3fbec3747309803bc86cbe42d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.895012 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 6c7e65dad084440787754e24cdd579d9 (DEFAULT_TABLE table=default.Table_511cd4142c9642d98428697731daef7a [id=c3c9a1fa404a4d1684a2edc8b13888f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.896076 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 6c7e65dad084440787754e24cdd579d9 (DEFAULT_TABLE table=default.Table_511cd4142c9642d98428697731daef7a [id=c3c9a1fa404a4d1684a2edc8b13888f4]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:46.897318 1128 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4251bbae62cf4646b2b390fec137e924 has no consensus state
I20260501 13:57:46.902711 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_3b1e200ad2674105a1911b12abed2310"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.904109 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b1e200ad2674105a1911b12abed2310 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:46.905855 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 6c7e65dad084440787754e24cdd579d9 (DEFAULT_TABLE table=default.Table_511cd4142c9642d98428697731daef7a [id=c3c9a1fa404a4d1684a2edc8b13888f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.906708 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2e6821c81b4eb2bf8cc283db989515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.907294 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c7e65dad084440787754e24cdd579d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.910902 2582 ts_tablet_manager.cc:1929] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:46.911306 2582 log.cc:1199] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/0669a229d6d54eacac9889ad18e06f30
I20260501 13:57:46.911720 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c7e65dad084440787754e24cdd579d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.911931 2582 ts_tablet_manager.cc:1950] T 0669a229d6d54eacac9889ad18e06f30 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:46.919739 1126 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 406e1af8b7d9488f88697206daa87c05 (table default.Table_759f3009542147de94c73784e1698246 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]). Expected version 1 got 0
I20260501 13:57:46.921849 1882 tablet_bootstrap.cc:492] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:46.924580 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbb10f2c3d864931abb3ae2a88372d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.925150 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05cfbdc8bc174f9698a5aa9a8e30ca04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.925740 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2e6821c81b4eb2bf8cc283db989515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.926446 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c7e65dad084440787754e24cdd579d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.930303 1754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2e6821c81b4eb2bf8cc283db989515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.949191 1601 tablet_service.cc:1511] Processing CreateTablet for tablet c20d4bb766404bb78ad87537eebc52c4 (DEFAULT_TABLE table=default.Table_34d3d9a3b69049028bd2692789a1a798 [id=95c19df4a8ed4e72b9a5d89e5ab67147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.953750 1462 tablet_service.cc:1511] Processing CreateTablet for tablet c20d4bb766404bb78ad87537eebc52c4 (DEFAULT_TABLE table=default.Table_34d3d9a3b69049028bd2692789a1a798 [id=95c19df4a8ed4e72b9a5d89e5ab67147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.954986 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c20d4bb766404bb78ad87537eebc52c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:46.956552 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_3b2f8dac784c40d0ae02c8d1fcd3a51e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:46.957957 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b2f8dac784c40d0ae02c8d1fcd3a51e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:46.959467 1129 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_76959fdf8538441aa44f9e120288cdab" } modify_external_catalogs: true
I20260501 13:57:46.959959 1129 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_76959fdf8538441aa44f9e120288cdab" } modify_external_catalogs: true
I20260501 13:57:46.962610 2623 raft_consensus.cc:493] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:46.963027 2623 raft_consensus.cc:515] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:46.964808 2623 leader_election.cc:290] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:46.966212 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cd353ca044465a6692656412e7d82" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:46.967383 1717 leader_election.cc:343] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:46.968889 1883 ts_tablet_manager.cc:1434] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.155s user 0.007s sys 0.000s
I20260501 13:57:46.980630 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet e8efee17e2c34caf98c0ede01d2a943a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:43 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.988476 1129 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 069f9e9b9b6c40d48c9a622b73e83dec
I20260501 13:57:46.990587 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet 069f9e9b9b6c40d48c9a622b73e83dec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:46 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:46.991278 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet 069f9e9b9b6c40d48c9a622b73e83dec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:46 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:46.991803 1461 tablet_service.cc:1558] Processing DeleteTablet for tablet 069f9e9b9b6c40d48c9a622b73e83dec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:46 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:46.992112 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 069f9e9b9b6c40d48c9a622b73e83dec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 069f9e9b9b6c40d48c9a622b73e83dec already in progress: creating tablet
I20260501 13:57:46.994110 1882 tablet_bootstrap.cc:654] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:46.995445 2047 ts_tablet_manager.cc:1929] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:46.998673 2047 log.cc:1199] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/67b0675398d04be08bd29638b4ba2083
I20260501 13:57:46.998688 1763 tablet_service.cc:1511] Processing CreateTablet for tablet c20d4bb766404bb78ad87537eebc52c4 (DEFAULT_TABLE table=default.Table_34d3d9a3b69049028bd2692789a1a798 [id=95c19df4a8ed4e72b9a5d89e5ab67147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:46.999539 2047 ts_tablet_manager.cc:1950] T 67b0675398d04be08bd29638b4ba2083 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:46.999655 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c20d4bb766404bb78ad87537eebc52c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:47.000620 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 67b0675398d04be08bd29638b4ba2083 (table default.Table_ddd647c4319b431f9d4e874bb528755d [id=d4ac3b610db54cb8aae381cbc882d18e]) successfully deleted
I20260501 13:57:47.000886 2047 tablet_replica.cc:333] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:47.003991 2047 raft_consensus.cc:2243] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:47.004516 2047 raft_consensus.cc:2272] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:47.006858 2047 ts_tablet_manager.cc:1916] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.009693 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 8b1577b8649c42b898c58eef90dee1c3 (DEFAULT_TABLE table=default.Table_1adc3e421970491abbba47fa8fe9ab5e [id=b0bedf9ac0584acea2717b17cab26ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:47.013644 1883 tablet_bootstrap.cc:492] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:47.018918 1455 tablet_service.cc:1511] Processing CreateTablet for tablet 8b1577b8649c42b898c58eef90dee1c3 (DEFAULT_TABLE table=default.Table_1adc3e421970491abbba47fa8fe9ab5e [id=b0bedf9ac0584acea2717b17cab26ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:47.022814 1126 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c81546b1614841bebe7a00f6ef1515dc" } modify_external_catalogs: true
I20260501 13:57:47.023313 1126 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_c81546b1614841bebe7a00f6ef1515dc" } modify_external_catalogs: true
I20260501 13:57:47.025687 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1577b8649c42b898c58eef90dee1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:47.029309 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cd353ca044465a6692656412e7d82" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:47.030642 1719 leader_election.cc:343] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.031032 1719 leader_election.cc:304] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:47.032047 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 293) took 1988 ms. Trace:
I20260501 13:57:47.033217 2582 rpcz_store.cc:276] 0501 13:57:45.043576 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.044400 (+ 824us) service_pool.cc:224] Handling call
0501 13:57:46.631205 (+1586805us) ts_tablet_manager.cc:1151] Deleting tablet 0669a229d6d54eacac9889ad18e06f30
0501 13:57:46.631247 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:46.765211 (+133964us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:46.910777 (+145566us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:47.031985 (+121208us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1586332}
I20260501 13:57:47.033738 2623 raft_consensus.cc:2749] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:47.035619 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 0669a229d6d54eacac9889ad18e06f30 (table default.Table_641d4d711c4d49319daa2cb0cf2714c6 [id=535a372705e4460ab657255e3f1be962]) successfully deleted
I20260501 13:57:47.038686 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet f52ea8fb8ae0427d804001aed42cd67c with delete_type TABLET_DATA_DELETED (replaced by tablet 4802dafc3e6b4874898b7dee694fc76e) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:47.045786 1882 tablet_bootstrap.cc:492] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:47.046224 1882 ts_tablet_manager.cc:1403] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.125s user 0.011s sys 0.000s
I20260501 13:57:47.048800 1881 tablet_bootstrap.cc:492] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:47.049273 1881 ts_tablet_manager.cc:1403] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.257s user 0.008s sys 0.002s
I20260501 13:57:47.049223 1882 raft_consensus.cc:359] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.049818 1882 raft_consensus.cc:385] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:47.050067 1882 raft_consensus.cc:740] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:47.050730 1882 consensus_queue.cc:260] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.051805 1881 raft_consensus.cc:359] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.052459 1881 raft_consensus.cc:385] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:47.052749 1881 raft_consensus.cc:740] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:47.053419 1881 consensus_queue.cc:260] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.053830 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c20d4bb766404bb78ad87537eebc52c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:47.060778 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1577b8649c42b898c58eef90dee1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:47.066227 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 8b1577b8649c42b898c58eef90dee1c3 (DEFAULT_TABLE table=default.Table_1adc3e421970491abbba47fa8fe9ab5e [id=b0bedf9ac0584acea2717b17cab26ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:47.068313 1132 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:47.071218 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 297) took 1980 ms. Trace:
I20260501 13:57:47.071576 2582 rpcz_store.cc:276] 0501 13:57:45.090528 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.090642 (+ 114us) service_pool.cc:224] Handling call
0501 13:57:47.034567 (+1943925us) ts_tablet_manager.cc:1151] Deleting tablet 2b7e55c79aa74ee6b164758daf6325fa
0501 13:57:47.070853 (+ 36286us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.071161 (+ 308us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1943311}
I20260501 13:57:47.072566 1883 tablet_bootstrap.cc:654] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:47.072856 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 298) took 1982 ms. Trace:
I20260501 13:57:47.073194 2582 rpcz_store.cc:276] 0501 13:57:45.112951 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.114653 (+ 1702us) service_pool.cc:224] Handling call
0501 13:57:47.072514 (+1957861us) ts_tablet_manager.cc:1151] Deleting tablet 3b0ed592fd3642c194bb946c8c6003b4
0501 13:57:47.072558 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.072808 (+ 250us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1957331}
I20260501 13:57:47.077993 1881 ts_tablet_manager.cc:1434] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20260501 13:57:47.082026 1129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_5e24fbb6bd82479b9f7ff4911a582bd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:47.083492 1129 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e24fbb6bd82479b9f7ff4911a582bd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:47.084182 1882 ts_tablet_manager.cc:1434] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20260501 13:57:47.087093 2564 tablet_replica.cc:333] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:47.087815 2564 raft_consensus.cc:2243] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:47.088253 2564 raft_consensus.cc:2272] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:47.090266 1882 tablet_bootstrap.cc:492] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:47.091347 2564 ts_tablet_manager.cc:1916] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.096596 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1577b8649c42b898c58eef90dee1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:47.102223 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_66a4e496efa34deaa6c793a210a510ff" } new_table_name: "default.Table_e2a480ab55e041c5a1adc51e98337db5" modify_external_catalogs: true
I20260501 13:57:47.102511 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_305c654ad4204e5d93626ca91d5731ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:47.103905 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_305c654ad4204e5d93626ca91d5731ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:47.107605 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_1adc3e421970491abbba47fa8fe9ab5e" } new_table_name: "default.Table_586f40c4fc464137a1cc94cac7cac38e" modify_external_catalogs: true
I20260501 13:57:47.074265 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 299) took 1937 ms. Trace:
I20260501 13:57:47.108310 2582 rpcz_store.cc:276] 0501 13:57:45.137028 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.162531 (+ 25503us) service_pool.cc:224] Handling call
0501 13:57:47.073971 (+1911440us) ts_tablet_manager.cc:1151] Deleting tablet 62d7ad7aa5ec4f79a2294023e17766d2
0501 13:57:47.074007 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.074220 (+ 213us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1910931}
W20260501 13:57:47.109194 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 2b7e55c79aa74ee6b164758daf6325fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b7e55c79aa74ee6b164758daf6325fa already in progress: creating tablet
W20260501 13:57:47.109807 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 3b0ed592fd3642c194bb946c8c6003b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b0ed592fd3642c194bb946c8c6003b4 already in progress: creating tablet
I20260501 13:57:47.111454 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 308) took 1655 ms. Trace:
I20260501 13:57:47.112099 2582 rpcz_store.cc:276] 0501 13:57:45.456478 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.456569 (+ 91us) service_pool.cc:224] Handling call
0501 13:57:47.110694 (+1654125us) ts_tablet_manager.cc:1151] Deleting tablet ef622e34d46a4b19a59e916d22c2b18b
0501 13:57:47.110776 (+ 82us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.111368 (+ 592us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1653604}
W20260501 13:57:47.114718 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet ef622e34d46a4b19a59e916d22c2b18b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef622e34d46a4b19a59e916d22c2b18b already in progress: creating tablet
I20260501 13:57:47.120061 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 307) took 1664 ms. Trace:
I20260501 13:57:47.130561 2582 rpcz_store.cc:276] 0501 13:57:45.456189 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.457218 (+ 1029us) service_pool.cc:224] Handling call
0501 13:57:47.118929 (+1661711us) ts_tablet_manager.cc:1151] Deleting tablet 71ac54444a414e25bcd05452edfbf5ba
0501 13:57:47.119712 (+ 783us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.120001 (+ 289us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1661169}
W20260501 13:57:47.133183 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 71ac54444a414e25bcd05452edfbf5ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71ac54444a414e25bcd05452edfbf5ba already in progress: creating tablet
I20260501 13:57:47.134315 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 309) took 1384 ms. Trace:
I20260501 13:57:47.136591 1881 tablet_bootstrap.cc:492] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:47.142175 2582 rpcz_store.cc:276] 0501 13:57:45.749470 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.750524 (+ 1054us) service_pool.cc:224] Handling call
0501 13:57:47.133957 (+1383433us) ts_tablet_manager.cc:1151] Deleting tablet 6399ac6a0d044ad2aed95ba30e6659b8
0501 13:57:47.134008 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.134259 (+ 251us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1382887}
W20260501 13:57:47.143397 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 6399ac6a0d044ad2aed95ba30e6659b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6399ac6a0d044ad2aed95ba30e6659b8 already in progress: creating tablet
I20260501 13:57:47.146301 2582 tablet_replica.cc:333] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:47.147585 2520 raft_consensus.cc:493] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:47.139094 1882 tablet_bootstrap.cc:654] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:47.148114 2520 raft_consensus.cc:515] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.151332 2520 leader_election.cc:290] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:47.152446 1563 leader_election.cc:343] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 69626ed095bd4df7859563c176dafffc
I20260501 13:57:47.152729 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69626ed095bd4df7859563c176dafffc" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:47.154034 1564 leader_election.cc:343] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.154431 1564 leader_election.cc:304] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:47.155483 2520 raft_consensus.cc:2749] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:47.156798 2582 raft_consensus.cc:2243] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:47.158816 2582 raft_consensus.cc:2272] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:47.172981 1130 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:47.183061 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:33 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:47.191960 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_641d4d711c4d49319daa2cb0cf2714c6" } modify_external_catalogs: true
I20260501 13:57:47.192564 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_641d4d711c4d49319daa2cb0cf2714c6" } modify_external_catalogs: true
W20260501 13:57:47.198210 1128 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef191926320a4e0fa8bd94e47e9b9a86 consensus state has no leader
I20260501 13:57:47.200659 1883 tablet_bootstrap.cc:492] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:47.200570 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_72bc40e077c7414c96936f5aa8eb7511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:47.201316 1883 ts_tablet_manager.cc:1403] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.188s user 0.009s sys 0.000s
W20260501 13:57:47.201979 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72bc40e077c7414c96936f5aa8eb7511 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:47.204772 1128 master_service.cc:954] ReplaceTablet: received request to replace tablet bdc412d7b5634ef2893b59467be4fa92 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:47.207944 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 6df58010b9084f4ab4710dd8eb5c4041
I20260501 13:57:47.209311 1456 tablet_service.cc:1558] Processing DeleteTablet for tablet 6df58010b9084f4ab4710dd8eb5c4041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:47.211566 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 6df58010b9084f4ab4710dd8eb5c4041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:47.211759 1883 raft_consensus.cc:359] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.222597 1883 raft_consensus.cc:385] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:47.222968 1883 raft_consensus.cc:740] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:47.223762 1883 consensus_queue.cc:260] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.218307 2047 ts_tablet_manager.cc:1929] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:47.230221 2047 log.cc:1199] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/0669a229d6d54eacac9889ad18e06f30
I20260501 13:57:47.230967 2047 ts_tablet_manager.cc:1950] T 0669a229d6d54eacac9889ad18e06f30 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:47.232056 2047 tablet_replica.cc:333] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:47.232867 2047 raft_consensus.cc:2243] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:47.233563 2047 raft_consensus.cc:2272] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:47.235033 1127 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 81230beeff8d434aa2dac467c35d9703
I20260501 13:57:47.241369 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_5c18f34cdebc4aba990d32e590b96317"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:47.242595 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 81230beeff8d434aa2dac467c35d9703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:47.246634 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c18f34cdebc4aba990d32e590b96317 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:47.251732 1883 ts_tablet_manager.cc:1434] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.050s user 0.001s sys 0.003s
I20260501 13:57:47.252607 1128 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet bdc412d7b5634ef2893b59467be4fa92
I20260501 13:57:47.253795 1128 catalog_manager.cc:6741] ReplaceTablet: tablet bdc412d7b5634ef2893b59467be4fa92 deleted and replaced by tablet 20e288c811ee423cb13d10cabaaf5ece
W20260501 13:57:47.266242 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b1577b8649c42b898c58eef90dee1c3 consensus state has no leader
I20260501 13:57:47.267333 1883 tablet_bootstrap.cc:492] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:47.274355 1601 tablet_service.cc:1558] Processing DeleteTablet for tablet 81230beeff8d434aa2dac467c35d9703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:47.275025 1605 tablet_service.cc:1558] Processing DeleteTablet for tablet bdc412d7b5634ef2893b59467be4fa92 with delete_type TABLET_DATA_DELETED (replaced by tablet 20e288c811ee423cb13d10cabaaf5ece) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:47.276741 2582 ts_tablet_manager.cc:1916] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.273932 2644 raft_consensus.cc:493] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:47.278615 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 6df58010b9084f4ab4710dd8eb5c4041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:47.278940 2644 raft_consensus.cc:515] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.280687 2644 leader_election.cc:290] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:47.283080 1563 leader_election.cc:343] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: a2cf9528d5c24fc68e3dfdd93bf5c7bb
I20260501 13:57:47.274936 1756 tablet_service.cc:1558] Processing DeleteTablet for tablet bdc412d7b5634ef2893b59467be4fa92 with delete_type TABLET_DATA_DELETED (replaced by tablet 20e288c811ee423cb13d10cabaaf5ece) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:47.290591 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 2c8d5d04a19443cfaa47b733b1d6ada6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:35 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:47.277637 1883 tablet_bootstrap.cc:654] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:47.240586 1456 tablet_service.cc:1558] Processing DeleteTablet for tablet 81230beeff8d434aa2dac467c35d9703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:47.256728 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet bdc412d7b5634ef2893b59467be4fa92 with delete_type TABLET_DATA_DELETED (replaced by tablet 20e288c811ee423cb13d10cabaaf5ece) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:47.261729 1881 tablet_bootstrap.cc:654] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:47.286247 1480 raft_consensus.cc:3060] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:47.287815 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2cf9528d5c24fc68e3dfdd93bf5c7bb" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:47.316293 2623 raft_consensus.cc:493] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:47.316787 2623 raft_consensus.cc:515] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.319773 2623 leader_election.cc:290] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:47.321506 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56cee07054a64e199a0bfc26b63eca2a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:47.322722 1883 tablet_bootstrap.cc:492] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
W20260501 13:57:47.322722 1717 leader_election.cc:343] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.323235 1883 ts_tablet_manager.cc:1403] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.002s
I20260501 13:57:47.324759 2582 ts_tablet_manager.cc:1929] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:47.325466 1883 raft_consensus.cc:359] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.326058 1883 raft_consensus.cc:385] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:47.325091 1882 tablet_bootstrap.cc:492] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:47.326303 1883 raft_consensus.cc:740] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:47.326869 1883 consensus_queue.cc:260] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.327517 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56cee07054a64e199a0bfc26b63eca2a" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:47.326630 1882 ts_tablet_manager.cc:1403] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.242s user 0.013s sys 0.000s
W20260501 13:57:47.328745 1719 leader_election.cc:343] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.332263 1719 leader_election.cc:304] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:47.333206 2623 raft_consensus.cc:2749] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:47.334323 1882 raft_consensus.cc:359] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.334972 1882 raft_consensus.cc:385] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:47.335211 1882 raft_consensus.cc:740] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:47.336815 1130 master_service.cc:954] ReplaceTablet: received request to replace tablet 3d0fa1a12a554984a893ec75e8b32382 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:47.326212 2582 log.cc:1199] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/e1d7a80ad4c4455a9ddee7a955e32635
I20260501 13:57:47.342715 2047 ts_tablet_manager.cc:1916] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.343287 2582 ts_tablet_manager.cc:1950] T e1d7a80ad4c4455a9ddee7a955e32635 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:47.348712 1883 ts_tablet_manager.cc:1434] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260501 13:57:47.354526 1883 tablet_bootstrap.cc:492] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:47.357707 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 311) took 1534 ms. Trace:
I20260501 13:57:47.256328 2564 ts_tablet_manager.cc:1929] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:47.360622 2564 log.cc:1199] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/f52ea8fb8ae0427d804001aed42cd67c
I20260501 13:57:47.361287 2564 ts_tablet_manager.cc:1950] T f52ea8fb8ae0427d804001aed42cd67c P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:47.363641 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet f52ea8fb8ae0427d804001aed42cd67c (table default.Table_fb6bb57c871e4855aba95f3b430a3fc7 [id=09adc59479f44d358ed186c100b690b8]) successfully deleted
I20260501 13:57:47.335753 1882 consensus_queue.cc:260] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:47.365113 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 6df58010b9084f4ab4710dd8eb5c4041 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6df58010b9084f4ab4710dd8eb5c4041 already in progress: creating tablet
I20260501 13:57:47.367527 1882 ts_tablet_manager.cc:1434] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20260501 13:57:47.358027 2582 rpcz_store.cc:276] 0501 13:57:45.823523 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.823621 (+ 98us) service_pool.cc:224] Handling call
0501 13:57:47.146141 (+1322520us) ts_tablet_manager.cc:1151] Deleting tablet e1d7a80ad4c4455a9ddee7a955e32635
0501 13:57:47.146184 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.292694 (+146510us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:47.324646 (+ 31952us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:47.357646 (+ 33000us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1320903}
I20260501 13:57:47.371167 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet e1d7a80ad4c4455a9ddee7a955e32635 (table default.Table_cf80c30d6b014d56aab7e294515801d5 [id=16dcfdbf585e45b29a6cdc6442d0990b]) successfully deleted
W20260501 13:57:47.373493 2614 lock_manager.cc:585] Waited 2 seconds to obtain row lock on key '<redacted>' (key hash 9156742303377354975) tablet 00000000000000000000000000000000 cur holder 0x7b5800031500
I20260501 13:57:47.375118 1882 tablet_bootstrap.cc:492] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:47.375451 1883 tablet_bootstrap.cc:654] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:47.388073 2623 raft_consensus.cc:493] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:47.388545 2623 raft_consensus.cc:515] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.390406 1883 tablet_bootstrap.cc:492] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:47.390949 1883 ts_tablet_manager.cc:1403] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.000s
I20260501 13:57:47.391773 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f7db275918c41139cf9bf44bba001a7" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:47.393347 1883 raft_consensus.cc:359] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:47.393962 1717 leader_election.cc:343] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.394560 2582 tablet_replica.cc:333] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:47.390337 2623 leader_election.cc:290] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:47.394642 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f7db275918c41139cf9bf44bba001a7" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:47.395334 2582 raft_consensus.cc:2243] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:47.395747 2582 raft_consensus.cc:2272] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:47.395849 1719 leader_election.cc:343] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.396204 1719 leader_election.cc:304] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:47.398038 2623 raft_consensus.cc:2749] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:47.401783 1828 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:47.402199 2582 ts_tablet_manager.cc:1916] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.394033 1883 raft_consensus.cc:385] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:47.403780 1883 raft_consensus.cc:740] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:47.403993 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_5fba8e9c07514a4babeda53e157a580f" } new_table_name: "default.Table_923bb6fa9e4342c38e3e9c2e952d02d8" modify_external_catalogs: true
I20260501 13:57:47.404392 1883 consensus_queue.cc:260] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.406337 1883 ts_tablet_manager.cc:1434] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
W20260501 13:57:47.408474 1564 leader_election.cc:343] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.408840 1564 leader_election.cc:304] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:47.409780 2644 raft_consensus.cc:2749] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:47.412010 1129 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_74984006d27c461b927ebb211fffc155" } modify_external_catalogs: true
I20260501 13:57:47.412499 1129 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_74984006d27c461b927ebb211fffc155" } modify_external_catalogs: true
I20260501 13:57:47.421136 1881 tablet_bootstrap.cc:492] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:47.421432 1882 tablet_bootstrap.cc:654] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:47.421782 1881 ts_tablet_manager.cc:1403] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.286s user 0.012s sys 0.000s
I20260501 13:57:47.423710 1883 tablet_bootstrap.cc:492] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:47.428785 1883 tablet_bootstrap.cc:654] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:47.431859 2582 ts_tablet_manager.cc:1929] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:47.432194 2582 log.cc:1199] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/39cd02738e5f4ecfbfa06d5541614ffd
I20260501 13:57:47.432758 2582 ts_tablet_manager.cc:1950] T 39cd02738e5f4ecfbfa06d5541614ffd P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
W20260501 13:57:47.437391 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 962f5ac4faef48ca9aa3fd09d35e5274 consensus state has no leader
I20260501 13:57:47.439246 1128 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_f3ef319b6f34419d987e039dd1c2eb54" } modify_external_catalogs: true
I20260501 13:57:47.439739 1128 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_f3ef319b6f34419d987e039dd1c2eb54" } modify_external_catalogs: true
W20260501 13:57:47.445199 1119 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 8b1577b8649c42b898c58eef90dee1c3 (table default.Table_586f40c4fc464137a1cc94cac7cac38e [id=b0bedf9ac0584acea2717b17cab26ace], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8b1577b8649c42b898c58eef90dee1c3 consensus state has no leader
I20260501 13:57:47.456240 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_ccc1648285ff4873a78a3ee0632630d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:47.457712 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccc1648285ff4873a78a3ee0632630d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:47.460002 2658 raft_consensus.cc:493] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f90b7ff170b4eadb27abc9ef866ffcf)
I20260501 13:57:47.460456 2658 raft_consensus.cc:515] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.471941 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_66ca9758ef054c48bc7e719d382f285e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:47.473417 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66ca9758ef054c48bc7e719d382f285e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:47.474864 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_c79a50afdbec462cb118e9f092e962f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:47.476195 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c79a50afdbec462cb118e9f092e962f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:47.476702 2623 raft_consensus.cc:493] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:47.477133 2623 raft_consensus.cc:515] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.478945 2623 leader_election.cc:290] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:47.480341 1123 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_200f07afb8cf4eafbace88f5cd3090d2" } new_table_name: "default.Table_6097a4730b784d39885bff5f7673d66f" modify_external_catalogs: true
I20260501 13:57:47.480954 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "301fad71fe6e4e53a36216c41f2850bf" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:47.482076 1719 leader_election.cc:343] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.488790 1127 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:47.490518 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "301fad71fe6e4e53a36216c41f2850bf" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:47.491281 2658 leader_election.cc:290] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:47.493891 1717 leader_election.cc:343] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.494243 1717 leader_election.cc:304] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:47.495283 2623 raft_consensus.cc:2749] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:47.495314 1779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33a355897309420c97b1257afaefe8d1" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:47.499758 1128 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet fc6b0cfd20174d7680f8981bb850c50f
I20260501 13:57:47.502789 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet fc6b0cfd20174d7680f8981bb850c50f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:47.503454 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 81230beeff8d434aa2dac467c35d9703 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81230beeff8d434aa2dac467c35d9703 already in progress: creating tablet
W20260501 13:57:47.504907 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet bdc412d7b5634ef2893b59467be4fa92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdc412d7b5634ef2893b59467be4fa92 already in progress: creating tablet
I20260501 13:57:47.504937 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet fc6b0cfd20174d7680f8981bb850c50f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:47.505721 2564 tablet_replica.cc:333] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:47.511610 1881 raft_consensus.cc:359] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.512451 1881 raft_consensus.cc:385] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:47.512746 1881 raft_consensus.cc:740] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:47.513532 1881 consensus_queue.cc:260] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.520390 2564 raft_consensus.cc:2243] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:47.525847 2564 raft_consensus.cc:2272] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:47.530506 1123 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd5d1afb88d14ddeb09a1c4d28324a35 consensus state has no leader
I20260501 13:57:47.530416 1128 catalog_manager.cc:5763] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:47.544575 1127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_17529a8d7af544bab22d0471bfea0f0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:47.546185 1127 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17529a8d7af544bab22d0471bfea0f0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:47.566502 1881 ts_tablet_manager.cc:1434] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.144s user 0.005s sys 0.000s
I20260501 13:57:47.567849 1126 master_service.cc:954] ReplaceTablet: received request to replace tablet 3eb6d30bed2a410db8b62fdff54a5fac from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:47.575484 1881 tablet_bootstrap.cc:492] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:47.528337 2564 ts_tablet_manager.cc:1916] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.578202 1883 tablet_bootstrap.cc:492] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:47.578640 1883 ts_tablet_manager.cc:1403] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.155s user 0.030s sys 0.038s
I20260501 13:57:47.586585 1881 tablet_bootstrap.cc:654] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:47.593264 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33a355897309420c97b1257afaefe8d1" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:47.601096 1532 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:47.613430 1883 raft_consensus.cc:359] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.617046 2047 ts_tablet_manager.cc:1929] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:47.618835 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet fc6b0cfd20174d7680f8981bb850c50f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:47.620620 1883 raft_consensus.cc:385] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:47.621696 1883 raft_consensus.cc:740] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:47.624094 1883 consensus_queue.cc:260] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.625351 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:47.631064 2047 log.cc:1199] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/ec4bd4cb1e70472aa09fc93afc3b1501
I20260501 13:57:47.631811 2047 ts_tablet_manager.cc:1950] T ec4bd4cb1e70472aa09fc93afc3b1501 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:47.633621 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 310) took 1077 ms. Trace:
I20260501 13:57:47.634016 2047 rpcz_store.cc:276] 0501 13:57:46.555930 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.556030 (+ 100us) service_pool.cc:224] Handling call
0501 13:57:47.231920 (+675890us) ts_tablet_manager.cc:1151] Deleting tablet ec4bd4cb1e70472aa09fc93afc3b1501
0501 13:57:47.231957 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.569849 (+337892us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:47.616922 (+ 47073us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:47.633564 (+ 16642us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":675391}
I20260501 13:57:47.634572 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 314) took 1807 ms. Trace:
I20260501 13:57:47.635196 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet ec4bd4cb1e70472aa09fc93afc3b1501 (table default.Table_6ee51a9fc8ae4340b14c506cf9c450c2 [id=e61fe8a70f66497db17db0afacc35100]) successfully deleted
I20260501 13:57:47.634912 2582 rpcz_store.cc:276] 0501 13:57:45.827504 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.828200 (+ 696us) service_pool.cc:224] Handling call
0501 13:57:47.374409 (+1546209us) ts_tablet_manager.cc:1151] Deleting tablet 39cd02738e5f4ecfbfa06d5541614ffd
0501 13:57:47.394406 (+ 19997us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.417387 (+ 22981us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:47.431735 (+ 14348us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:47.634514 (+202779us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23063,"tablet-delete.queue_time_us":1545701}
I20260501 13:57:47.635972 2582 tablet_replica.cc:333] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:47.636647 2582 raft_consensus.cc:2243] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:47.637053 2582 raft_consensus.cc:2272] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:47.645582 2582 ts_tablet_manager.cc:1916] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.649156 1883 ts_tablet_manager.cc:1434] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.070s user 0.006s sys 0.000s
I20260501 13:57:47.655079 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 313) took 1037 ms. Trace:
I20260501 13:57:47.655400 2047 rpcz_store.cc:276] 0501 13:57:46.617775 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.617871 (+ 96us) service_pool.cc:224] Handling call
0501 13:57:47.654661 (+1036790us) ts_tablet_manager.cc:1151] Deleting tablet a000b129a6ab429889b543c20750a493
0501 13:57:47.654701 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.655018 (+ 317us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018214}
W20260501 13:57:47.656905 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet a000b129a6ab429889b543c20750a493 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a000b129a6ab429889b543c20750a493 already in progress: creating tablet
W20260501 13:57:47.657612 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet bd633c026195429ca69d8041809bee45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd633c026195429ca69d8041809bee45 already in progress: creating tablet
I20260501 13:57:47.657758 2564 ts_tablet_manager.cc:1929] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:47.658138 2564 log.cc:1199] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/2c8d5d04a19443cfaa47b733b1d6ada6
W20260501 13:57:47.660058 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 37f57009c05f479481b59e49062608d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37f57009c05f479481b59e49062608d5 already in progress: creating tablet
I20260501 13:57:47.666869 2047 tablet_replica.cc:333] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:47.667626 2047 raft_consensus.cc:2243] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:47.668098 2047 raft_consensus.cc:2272] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:47.664331 1883 tablet_bootstrap.cc:492] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:47.674331 2520 raft_consensus.cc:493] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:47.675443 2520 raft_consensus.cc:515] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.658768 2564 ts_tablet_manager.cc:1950] T 2c8d5d04a19443cfaa47b733b1d6ada6 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:47.681157 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 2c8d5d04a19443cfaa47b733b1d6ada6 (table default.Table_cc6986b178ed47349a3dfda0ef76ce70 [id=b0864c2d4d764c42ba3ec7a2bffc13b6]) successfully deleted
I20260501 13:57:47.683593 2564 tablet_replica.cc:333] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:47.684224 1881 tablet_bootstrap.cc:492] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:47.684310 2564 raft_consensus.cc:2243] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:47.684191 1128 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_be2275f0abfb43abb3bd0b1e128d7fef" } new_table_name: "default.Table_12bdb1ff2b634a3fae05d8eb2bcd6c4d" modify_external_catalogs: true
I20260501 13:57:47.684608 1881 ts_tablet_manager.cc:1403] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.007s
I20260501 13:57:47.682994 2047 ts_tablet_manager.cc:1916] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.686826 1881 raft_consensus.cc:359] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.687420 1881 raft_consensus.cc:385] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:47.687675 1881 raft_consensus.cc:740] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:47.689031 1882 tablet_bootstrap.cc:492] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:47.689469 1882 ts_tablet_manager.cc:1403] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.315s user 0.017s sys 0.003s
I20260501 13:57:47.690865 2564 raft_consensus.cc:2272] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:47.691493 2520 leader_election.cc:290] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:47.692725 2564 ts_tablet_manager.cc:1916] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.693312 1421 leader_election.cc:304] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:47.694512 2678 raft_consensus.cc:2749] T 33a355897309420c97b1257afaefe8d1 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:57:47.694940 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 99818.2us
I20260501 13:57:47.698817 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 39cd02738e5f4ecfbfa06d5541614ffd (table default.Table_6f9bd93befbf45c0a7415d32ce58d7fb [id=edb6ad6e0b444d3b8930c43ce4082a5c]) successfully deleted
I20260501 13:57:47.701876 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8efee17e2c34caf98c0ede01d2a943a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:47.703240 1126 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 3eb6d30bed2a410db8b62fdff54a5fac
I20260501 13:57:47.704403 1126 catalog_manager.cc:6741] ReplaceTablet: tablet 3eb6d30bed2a410db8b62fdff54a5fac deleted and replaced by tablet 352c10a8418c4ac4a3caaef3c08ef07a
I20260501 13:57:47.705238 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 3eb6d30bed2a410db8b62fdff54a5fac with delete_type TABLET_DATA_DELETED (replaced by tablet 352c10a8418c4ac4a3caaef3c08ef07a) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:47.706889 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet 3eb6d30bed2a410db8b62fdff54a5fac with delete_type TABLET_DATA_DELETED (replaced by tablet 352c10a8418c4ac4a3caaef3c08ef07a) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:47.710737 1881 consensus_queue.cc:260] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.713295 1127 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ca507f190ca9413d91939e27911c79d1" } modify_external_catalogs: true
I20260501 13:57:47.713780 1127 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_ca507f190ca9413d91939e27911c79d1" } modify_external_catalogs: true
I20260501 13:57:47.723199 1602 tablet_service.cc:1558] Processing DeleteTablet for tablet 3eb6d30bed2a410db8b62fdff54a5fac with delete_type TABLET_DATA_DELETED (replaced by tablet 352c10a8418c4ac4a3caaef3c08ef07a) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:47.725384 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet 38c3eb388d474d718554775d65f58d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:47.728806 1882 raft_consensus.cc:359] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.729558 1882 raft_consensus.cc:385] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:47.729857 1882 raft_consensus.cc:740] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:47.730504 1882 consensus_queue.cc:260] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.734119 1882 ts_tablet_manager.cc:1434] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.044s user 0.002s sys 0.003s
I20260501 13:57:47.739233 1882 tablet_bootstrap.cc:492] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:47.739569 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_b29786a52abd4fad9a2337d7c76fbbc4" } new_table_name: "default.Table_661669634c2e491d8b6d43bb97c1e40f" modify_external_catalogs: true
I20260501 13:57:47.749091 1883 tablet_bootstrap.cc:654] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:47.741048 1564 leader_election.cc:343] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.745822 1675 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38815
I20260501 13:57:47.743511 1881 ts_tablet_manager.cc:1434] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.059s user 0.006s sys 0.001s
I20260501 13:57:47.756239 1881 tablet_bootstrap.cc:492] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:47.753628 2582 ts_tablet_manager.cc:1929] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:47.758957 2582 log.cc:1199] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/67b0675398d04be08bd29638b4ba2083
I20260501 13:57:47.762773 2582 ts_tablet_manager.cc:1950] T 67b0675398d04be08bd29638b4ba2083 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:47.763557 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_a56a7addc3cd4e4aab56ab751ddcb8c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:47.765658 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a56a7addc3cd4e4aab56ab751ddcb8c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:47.769467 2047 ts_tablet_manager.cc:1929] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:47.769876 2047 log.cc:1199] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/d361a2d82ecd456fb43f5e7fe3c83094
W20260501 13:57:47.774411 1564 consensus_peers.cc:597] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:47.781762 2686 raft_consensus.cc:493] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:47.782222 2686 raft_consensus.cc:515] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.784070 2686 leader_election.cc:290] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:47.786230 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9169851f1358460d99a48cb6067ab3e3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:47.788225 1779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9169851f1358460d99a48cb6067ab3e3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:47.788885 1779 raft_consensus.cc:2468] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:47.799968 2685 consensus_queue.cc:1048] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:57:47.787626 1564 leader_election.cc:343] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.803565 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 312) took 1976 ms. Trace:
I20260501 13:57:47.805486 1883 tablet_bootstrap.cc:492] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:47.805925 1883 ts_tablet_manager.cc:1403] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.142s user 0.004s sys 0.008s
I20260501 13:57:47.808643 1883 raft_consensus.cc:359] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.809301 1883 raft_consensus.cc:385] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:47.809564 1883 raft_consensus.cc:740] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:47.810232 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_c4a062dd073f477294baea6a237e0392"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:47.810237 1883 consensus_queue.cc:260] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:47.811796 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4a062dd073f477294baea6a237e0392 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:47.813357 1881 tablet_bootstrap.cc:654] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:47.804838 1563 leader_election.cc:343] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: e8efee17e2c34caf98c0ede01d2a943a
I20260501 13:57:47.815254 1563 leader_election.cc:304] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:47.816121 2685 raft_consensus.cc:2749] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:47.820168 1563 leader_election.cc:304] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:47.821012 2685 raft_consensus.cc:2804] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:47.821398 2685 raft_consensus.cc:493] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:47.821918 2685 raft_consensus.cc:3060] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:47.835850 1883 ts_tablet_manager.cc:1434] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260501 13:57:47.837034 2564 ts_tablet_manager.cc:1929] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:47.837399 2564 log.cc:1199] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/fc6b0cfd20174d7680f8981bb850c50f
I20260501 13:57:47.838096 2564 ts_tablet_manager.cc:1950] T fc6b0cfd20174d7680f8981bb850c50f P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:47.841207 1883 tablet_bootstrap.cc:492] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:47.844290 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_8722bc86cc204923b96c371a00b62e75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:47.844385 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_1b0b21c8b69f49239f2fdeeac464d4ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:47.845876 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b0b21c8b69f49239f2fdeeac464d4ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:47.845932 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8722bc86cc204923b96c371a00b62e75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:47.846901 2582 rpcz_store.cc:276] 0501 13:57:45.826932 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.830620 (+ 3688us) service_pool.cc:224] Handling call
0501 13:57:47.635812 (+1805192us) ts_tablet_manager.cc:1151] Deleting tablet 67b0675398d04be08bd29638b4ba2083
0501 13:57:47.635852 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.669409 (+ 33557us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:47.753506 (+ 84097us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:47.803506 (+ 50000us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2693,"tablet-delete.queue_time_us":1804701}
I20260501 13:57:47.853595 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 67b0675398d04be08bd29638b4ba2083 (table default.Table_ddd647c4319b431f9d4e874bb528755d [id=d4ac3b610db54cb8aae381cbc882d18e]) successfully deleted
I20260501 13:57:47.854157 2582 tablet_replica.cc:333] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:47.854915 2582 raft_consensus.cc:2243] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:47.814841 1882 tablet_bootstrap.cc:654] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:47.862218 2685 raft_consensus.cc:515] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:47.864487 2564 tablet_replica.cc:333] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:47.864466 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet fc6b0cfd20174d7680f8981bb850c50f (table default.Table_f3ef319b6f34419d987e039dd1c2eb54 [id=c7df1385355c46c0896e9586953edf34]) successfully deleted
I20260501 13:57:47.865199 2564 raft_consensus.cc:2243] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:47.865629 2564 raft_consensus.cc:2272] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:47.883438 2564 ts_tablet_manager.cc:1916] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.822664 2047 ts_tablet_manager.cc:1950] T d361a2d82ecd456fb43f5e7fe3c83094 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:47.912045 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 321) took 1035 ms. Trace:
I20260501 13:57:47.912376 2047 rpcz_store.cc:276] 0501 13:57:46.877107 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.877232 (+ 125us) service_pool.cc:224] Handling call
0501 13:57:47.666711 (+789479us) ts_tablet_manager.cc:1151] Deleting tablet d361a2d82ecd456fb43f5e7fe3c83094
0501 13:57:47.666751 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:47.724291 (+ 57540us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:47.769347 (+ 45056us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:47.911993 (+142646us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":788968}
I20260501 13:57:47.914114 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet d361a2d82ecd456fb43f5e7fe3c83094 (table default.Table_945a3cdaaa824caab261338095683b43 [id=e5c5a3c3a1544b599c53a9b56a6c7297]) successfully deleted
W20260501 13:57:47.915253 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 069f9e9b9b6c40d48c9a622b73e83dec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 069f9e9b9b6c40d48c9a622b73e83dec already in progress: creating tablet
I20260501 13:57:47.915933 2047 tablet_replica.cc:333] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
W20260501 13:57:47.915892 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 6df58010b9084f4ab4710dd8eb5c4041 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6df58010b9084f4ab4710dd8eb5c4041 already in progress: creating tablet
I20260501 13:57:47.916769 2047 raft_consensus.cc:2243] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260501 13:57:47.917017 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 81230beeff8d434aa2dac467c35d9703 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81230beeff8d434aa2dac467c35d9703 already in progress: creating tablet
I20260501 13:57:47.917371 2047 raft_consensus.cc:2272] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:47.917577 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet bdc412d7b5634ef2893b59467be4fa92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdc412d7b5634ef2893b59467be4fa92 already in progress: creating tablet
I20260501 13:57:47.917420 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9169851f1358460d99a48cb6067ab3e3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
W20260501 13:57:47.918823 1564 leader_election.cc:343] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:47.919574 2047 ts_tablet_manager.cc:1916] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.887825 1883 tablet_bootstrap.cc:654] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:47.903764 2582 raft_consensus.cc:2272] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:47.936129 2582 ts_tablet_manager.cc:1916] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:47.939949 2685 leader_election.cc:290] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:47.941427 1781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9169851f1358460d99a48cb6067ab3e3" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:47.941978 1781 raft_consensus.cc:3060] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:47.959288 1781 raft_consensus.cc:2468] T 9169851f1358460d99a48cb6067ab3e3 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:47.979418 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_524cc370fa15424ca5fb7c094c345897" } new_table_name: "default.Table_e69b3d0935e64d3a9853c5dfab42e260" modify_external_catalogs: true
I20260501 13:57:47.990590 1883 tablet_bootstrap.cc:492] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:47.991068 1883 ts_tablet_manager.cc:1403] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.150s user 0.010s sys 0.003s
I20260501 13:57:47.999131 1563 leader_election.cc:304] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:48.003862 1882 tablet_bootstrap.cc:492] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:48.007287 1883 raft_consensus.cc:359] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.007908 1883 raft_consensus.cc:385] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.008165 1883 raft_consensus.cc:740] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:48.008705 1883 consensus_queue.cc:260] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.004563 2685 raft_consensus.cc:2804] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:48.013897 2623 raft_consensus.cc:493] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:48.014037 2685 raft_consensus.cc:697] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:48.014307 2623 raft_consensus.cc:515] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.014835 2685 consensus_queue.cc:237] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.017859 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31c73582788648aab55d678ad6e4fc76" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:48.019543 1719 leader_election.cc:343] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:48.016147 2623 leader_election.cc:290] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:48.022804 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31c73582788648aab55d678ad6e4fc76" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:48.025570 2564 ts_tablet_manager.cc:1929] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260501 13:57:48.025800 1717 leader_election.cc:343] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:48.025923 2564 log.cc:1199] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/3eb6d30bed2a410db8b62fdff54a5fac
I20260501 13:57:48.026165 1717 leader_election.cc:304] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:48.027194 2623 raft_consensus.cc:2749] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:48.028337 2710 tablet.cc:1724] T 96e937b3a0b5451e8eaf4d5c47c17620 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:48.030531 2711 tablet.cc:1724] T 96e937b3a0b5451e8eaf4d5c47c17620 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:48.031641 1123 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_d17e515649da406b814a3e991d8f0d41" } new_table_name: "default.Table_c8b1ea252e544b5f81a72380b00d0b0a" modify_external_catalogs: true
I20260501 13:57:48.032613 2709 tablet.cc:1724] T 1409fbe5846a4bb099f31fb2d2e41382 P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:48.035473 2708 tablet.cc:1724] T 1409fbe5846a4bb099f31fb2d2e41382 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:48.039081 1883 ts_tablet_manager.cc:1434] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.048s user 0.004s sys 0.002s
I20260501 13:57:48.044466 1881 tablet_bootstrap.cc:492] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:48.044942 1881 ts_tablet_manager.cc:1403] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.289s user 0.010s sys 0.000s
I20260501 13:57:48.007771 1882 ts_tablet_manager.cc:1403] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.269s user 0.006s sys 0.005s
W20260501 13:57:48.049015 1126 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2aca712596e440b797010df853137b16 consensus state has no leader
I20260501 13:57:48.044785 2047 ts_tablet_manager.cc:1929] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:48.052925 2047 log.cc:1199] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/fc6b0cfd20174d7680f8981bb850c50f
I20260501 13:57:48.054045 2047 ts_tablet_manager.cc:1950] T fc6b0cfd20174d7680f8981bb850c50f P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:48.060899 2715 tablet.cc:1724] T 1409fbe5846a4bb099f31fb2d2e41382 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:48.064705 2047 tablet_replica.cc:333] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:48.067452 1882 raft_consensus.cc:359] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.068117 1882 raft_consensus.cc:385] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.068369 1882 raft_consensus.cc:740] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:48.069001 1882 consensus_queue.cc:260] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.067905 1883 tablet_bootstrap.cc:492] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:48.069615 1123 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0468f84c6d1b4440b82faebd8a33fc9f consensus state has no leader
I20260501 13:57:48.070729 2564 ts_tablet_manager.cc:1950] T 3eb6d30bed2a410db8b62fdff54a5fac P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:48.073243 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 3eb6d30bed2a410db8b62fdff54a5fac (table default.Table_52acb3d1a0d1450bb41440f4de75ac91 [id=7b430735e8b640dc84cf92f308f20362]) successfully deleted
I20260501 13:57:48.073254 1881 raft_consensus.cc:359] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.073550 2714 tablet.cc:1724] T 96e937b3a0b5451e8eaf4d5c47c17620 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260501 13:57:48.073921 1881 raft_consensus.cc:385] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.074152 1881 raft_consensus.cc:740] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:48.066107 2047 raft_consensus.cc:2243] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:48.063941 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet fc6b0cfd20174d7680f8981bb850c50f (table default.Table_f3ef319b6f34419d987e039dd1c2eb54 [id=c7df1385355c46c0896e9586953edf34]) successfully deleted
I20260501 13:57:48.089432 2047 raft_consensus.cc:2272] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:48.091806 2047 ts_tablet_manager.cc:1916] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:48.092137 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_208156cfc4324941b36574ed9fb2c398"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:48.093604 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_208156cfc4324941b36574ed9fb2c398 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.099617 1883 tablet_bootstrap.cc:654] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:48.106129 1123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_70a73112cf374a6db6a56bcc5faf040d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:48.107872 1123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70a73112cf374a6db6a56bcc5faf040d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.112906 1881 consensus_queue.cc:260] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.147957 1881 ts_tablet_manager.cc:1434] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.103s user 0.005s sys 0.000s
I20260501 13:57:48.110280 1882 ts_tablet_manager.cc:1434] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.062s user 0.005s sys 0.000s
I20260501 13:57:48.149216 1882 tablet_bootstrap.cc:492] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
W20260501 13:57:48.155498 2699 log.cc:927] Time spent T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: Append to log took a long time: real 0.218s user 0.029s sys 0.007s
I20260501 13:57:48.158594 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_e095439dec874189b8a4ea72e4dc1b48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:48.191886 1131 master_service.cc:954] ReplaceTablet: received request to replace tablet 9d38d75fcd494fc4a3420bcf15f2c70f from {username='slave'} at 127.0.0.1:41700
W20260501 13:57:48.194123 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e095439dec874189b8a4ea72e4dc1b48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.208954 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_88ada069a99749fea12f89d823a4e95a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:48.210438 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88ada069a99749fea12f89d823a4e95a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.214558 1883 tablet_bootstrap.cc:492] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:48.214989 1883 ts_tablet_manager.cc:1403] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.175s user 0.007s sys 0.004s
I20260501 13:57:48.217221 1883 raft_consensus.cc:359] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.217803 1883 raft_consensus.cc:385] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.218036 1883 raft_consensus.cc:740] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:48.219281 1883 consensus_queue.cc:260] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.221207 1883 ts_tablet_manager.cc:1434] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260501 13:57:48.227536 1881 tablet_bootstrap.cc:492] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:48.249974 2685 raft_consensus.cc:493] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:48.250561 2685 raft_consensus.cc:515] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.256953 1777 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d399de014d74cab8df14f44787ca2f7" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:48.257534 1777 raft_consensus.cc:2468] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:48.258719 1563 leader_election.cc:304] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:48.259421 2644 raft_consensus.cc:2804] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:48.259763 2644 raft_consensus.cc:493] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:48.260073 2644 raft_consensus.cc:3060] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:48.262954 1126 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_fb33a28f96da45c29f6e68281ec90b8d" } new_table_name: "default.Table_1d2b18a2fcea4bb9b93217341cb81a97" modify_external_catalogs: true
I20260501 13:57:48.266796 2047 ts_tablet_manager.cc:1929] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:48.273957 2644 raft_consensus.cc:515] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.276765 2685 leader_election.cc:290] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:48.279458 2047 log.cc:1199] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/51ba7b8a7f4d4ec6ba4af775596d3b59
I20260501 13:57:48.286994 2047 ts_tablet_manager.cc:1950] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:48.288442 2644 leader_election.cc:290] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:48.289399 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 (table default.Table_7b8be07c742f48d0af3422dfea143dd1 [id=39aeda29815d474196136813b5d811e7]) successfully deleted
I20260501 13:57:48.290189 1777 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d399de014d74cab8df14f44787ca2f7" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:48.290733 1777 raft_consensus.cc:3060] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:48.299273 2623 raft_consensus.cc:493] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:48.299463 1882 tablet_bootstrap.cc:654] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:48.299757 2623 raft_consensus.cc:515] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.301692 2623 leader_election.cc:290] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:48.303162 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0468f84c6d1b4440b82faebd8a33fc9f" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:48.304903 1719 leader_election.cc:343] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:48.304793 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0468f84c6d1b4440b82faebd8a33fc9f" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:48.305979 1717 leader_election.cc:343] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:48.306325 1717 leader_election.cc:304] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:48.307291 2623 raft_consensus.cc:2749] T 0468f84c6d1b4440b82faebd8a33fc9f P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:48.309008 1777 raft_consensus.cc:2468] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:48.309588 2582 ts_tablet_manager.cc:1929] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:48.309968 2582 log.cc:1199] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/d361a2d82ecd456fb43f5e7fe3c83094
I20260501 13:57:48.310194 1563 leader_election.cc:304] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:48.310817 1883 tablet_bootstrap.cc:492] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:48.311069 2644 raft_consensus.cc:2804] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:48.311467 2644 raft_consensus.cc:697] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:48.312275 2644 consensus_queue.cc:237] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.319115 1123 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_12bdb1ff2b634a3fae05d8eb2bcd6c4d [id=8d39f1f13bc24a2c84e888bb29b6e74c] alter complete (version 2)
I20260501 13:57:48.321110 1883 tablet_bootstrap.cc:654] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:48.322724 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d399de014d74cab8df14f44787ca2f7" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:48.323133 2047 tablet_replica.cc:333] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:48.323896 2047 raft_consensus.cc:2243] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:48.324086 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d399de014d74cab8df14f44787ca2f7" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:48.324438 2047 raft_consensus.cc:2272] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:48.326514 2047 ts_tablet_manager.cc:1916] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:48.330518 1564 leader_election.cc:343] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:48.331593 1564 leader_election.cc:343] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:48.267594 1881 tablet_bootstrap.cc:654] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:48.339972 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_133c504302a64fe0b6133ad5cfe65de1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:48.341317 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_295132c6928c4b309137696f9015de9f" } new_table_name: "default.Table_02c2875fa5944b569e671e0f7719bffd" modify_external_catalogs: true
I20260501 13:57:48.344601 2582 ts_tablet_manager.cc:1950] T d361a2d82ecd456fb43f5e7fe3c83094 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:48.346161 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 316) took 2512 ms. Trace:
I20260501 13:57:48.346503 2582 rpcz_store.cc:276] 0501 13:57:45.833620 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:45.852329 (+ 18709us) service_pool.cc:224] Handling call
0501 13:57:47.854008 (+2001679us) ts_tablet_manager.cc:1151] Deleting tablet d361a2d82ecd456fb43f5e7fe3c83094
0501 13:57:47.854050 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:48.114122 (+260072us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:48.309467 (+195345us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:48.346084 (+ 36617us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60630,"tablet-delete.queue_time_us":1942663}
I20260501 13:57:48.347781 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet d361a2d82ecd456fb43f5e7fe3c83094 (table default.Table_945a3cdaaa824caab261338095683b43 [id=e5c5a3c3a1544b599c53a9b56a6c7297]) successfully deleted
I20260501 13:57:48.348696 2582 tablet_replica.cc:333] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:48.349509 2582 raft_consensus.cc:2243] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:48.350001 2582 raft_consensus.cc:2272] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:48.363473 1126 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71f7bd151fa64fe5a80e756a95a375c5 consensus state has no leader
I20260501 13:57:48.365587 1882 tablet_bootstrap.cc:492] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:48.366075 1882 ts_tablet_manager.cc:1403] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.217s user 0.009s sys 0.001s
W20260501 13:57:48.368268 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_133c504302a64fe0b6133ad5cfe65de1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.368660 1881 tablet_bootstrap.cc:492] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:48.368462 1882 raft_consensus.cc:359] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.369057 1881 ts_tablet_manager.cc:1403] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.142s user 0.009s sys 0.000s
I20260501 13:57:48.369045 1882 raft_consensus.cc:385] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.369285 1882 raft_consensus.cc:740] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:48.369894 1882 consensus_queue.cc:260] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.372907 2582 ts_tablet_manager.cc:1916] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:48.376827 1882 ts_tablet_manager.cc:1434] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260501 13:57:48.381214 2635 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f04a112d3be549d98b4ee77dbefacb7b)
I20260501 13:57:48.381831 2635 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:48.388389 2583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f04a112d3be549d98b4ee77dbefacb7b. Latest consensus state: current_term: 3 leader_uuid: "f04a112d3be549d98b4ee77dbefacb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:48.389052 2583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:48.390828 2702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f04a112d3be549d98b4ee77dbefacb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } } }
I20260501 13:57:48.391536 2702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:57:48.391242 1126 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_d22b3187808547e88c2cfb3f9b22c917"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:48.392695 1126 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d22b3187808547e88c2cfb3f9b22c917 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.396027 2623 raft_consensus.cc:493] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:48.396487 2623 raft_consensus.cc:515] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.399598 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab056da506f454984a07026341c534b" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:48.407099 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab056da506f454984a07026341c534b" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:48.398212 2623 leader_election.cc:290] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
W20260501 13:57:48.405745 1717 leader_election.cc:343] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:48.409094 1719 leader_election.cc:343] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:48.409435 1719 leader_election.cc:304] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:48.410660 2623 raft_consensus.cc:2749] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:48.436684 1882 tablet_bootstrap.cc:492] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:48.439637 1883 tablet_bootstrap.cc:492] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:48.440111 1883 ts_tablet_manager.cc:1403] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.129s user 0.008s sys 0.001s
I20260501 13:57:48.446138 2635 leader_election.cc:290] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897)
I20260501 13:57:48.453645 2485 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" instance_seqno: 1777643842271742) as {username='slave'} at 127.1.16.129:39017; Asking this server to re-register.
W20260501 13:57:48.455495 1119 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 8b1577b8649c42b898c58eef90dee1c3 (table default.Table_586f40c4fc464137a1cc94cac7cac38e [id=b0bedf9ac0584acea2717b17cab26ace], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8b1577b8649c42b898c58eef90dee1c3 consensus state has no leader
I20260501 13:57:48.455945 1881 raft_consensus.cc:359] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.456565 1881 raft_consensus.cc:385] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.456805 1881 raft_consensus.cc:740] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:48.457425 1881 consensus_queue.cc:260] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.460600 1532 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:48.465143 2502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40472 (request call id 0) took 3843 ms. Trace:
I20260501 13:57:48.472524 1882 tablet_bootstrap.cc:654] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:48.476336 1883 raft_consensus.cc:359] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.476962 1883 raft_consensus.cc:385] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.477219 1883 raft_consensus.cc:740] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:48.489923 1123 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_661669634c2e491d8b6d43bb97c1e40f [id=05998ead56ed454dab9a71ef2ac9cb58] alter complete (version 2)
I20260501 13:57:48.490161 2582 ts_tablet_manager.cc:1929] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:48.491639 1881 ts_tablet_manager.cc:1434] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.122s user 0.006s sys 0.000s
I20260501 13:57:48.491853 2623 raft_consensus.cc:493] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:48.492483 1881 tablet_bootstrap.cc:492] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:48.492300 2623 raft_consensus.cc:515] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:48.493750 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b2e6821c81b4eb2bf8cc283db989515 consensus state has no leader
I20260501 13:57:48.494143 2623 leader_election.cc:290] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:48.496402 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50ec3d3f3eb449759a8df52279b2e5e8" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:48.497346 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 with delete_type TABLET_DATA_DELETED (replaced by tablet f28db04617a449af8d87a554fb63e1e4) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:48.498576 1883 consensus_queue.cc:260] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.499060 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50ec3d3f3eb449759a8df52279b2e5e8" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:48.502768 1717 leader_election.cc:343] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:48.504179 1719 leader_election.cc:343] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:48.504539 1719 leader_election.cc:304] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:48.510221 2719 raft_consensus.cc:493] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:48.475924 2485 ts_manager.cc:194] Registered new tserver with Master: 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:48.510701 2719 raft_consensus.cc:515] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.513327 2719 leader_election.cc:290] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:48.514544 2623 raft_consensus.cc:2749] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:48.514370 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38c3eb388d474d718554775d65f58d78" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:48.515024 1626 raft_consensus.cc:2468] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 0.
W20260501 13:57:48.515146 1418 leader_election.cc:343] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 38c3eb388d474d718554775d65f58d78
I20260501 13:57:48.516360 1421 leader_election.cc:304] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:48.517639 2719 raft_consensus.cc:2804] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:48.521162 2719 raft_consensus.cc:493] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:48.521562 2719 raft_consensus.cc:3060] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:48.525792 1883 ts_tablet_manager.cc:1434] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.085s user 0.007s sys 0.000s
I20260501 13:57:48.528211 2719 raft_consensus.cc:515] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.526773 1883 tablet_bootstrap.cc:492] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:48.538781 2699 log.cc:927] Time spent T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260501 13:57:48.546903 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_8a2507a3f60d406fbb5cc29416579789"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:48.548352 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a2507a3f60d406fbb5cc29416579789 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.550952 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_f81cde0094724a1bb91d297f541f7390"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:48.552460 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f81cde0094724a1bb91d297f541f7390 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.576213 2720 raft_consensus.cc:493] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:48.576629 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38c3eb388d474d718554775d65f58d78" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:48.577144 1626 raft_consensus.cc:3060] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:48.577162 2720 raft_consensus.cc:515] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.579176 2720 leader_election.cc:290] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:48.580865 1777 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2ac522f2e4c4ab48a9b7d211c82ed9e" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:48.569217 2047 ts_tablet_manager.cc:1929] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:48.518702 1881 tablet_bootstrap.cc:654] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:48.583572 2582 log.cc:1199] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/94abe840b1994c0cb1f7ac7f4ac09cd9
I20260501 13:57:48.584008 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2ac522f2e4c4ab48a9b7d211c82ed9e" candidate_uuid: "993b9ba18d8547f8af43167e727a6a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:48.584290 2582 ts_tablet_manager.cc:1950] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:48.588231 1883 tablet_bootstrap.cc:654] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:48.588837 1626 raft_consensus.cc:2468] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993b9ba18d8547f8af43167e727a6a24 in term 1.
I20260501 13:57:48.591765 1882 tablet_bootstrap.cc:492] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:48.592181 1882 ts_tablet_manager.cc:1403] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.156s user 0.002s sys 0.007s
I20260501 13:57:48.585659 1421 leader_election.cc:304] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:48.599797 2719 leader_election.cc:290] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:48.604228 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 318) took 2182 ms. Trace:
I20260501 13:57:48.604425 2719 raft_consensus.cc:2749] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:48.601727 1418 leader_election.cc:343] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 38c3eb388d474d718554775d65f58d78
I20260501 13:57:48.604542 2582 rpcz_store.cc:276] 0501 13:57:46.421342 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.421447 (+ 105us) service_pool.cc:224] Handling call
0501 13:57:48.348545 (+1927098us) ts_tablet_manager.cc:1151] Deleting tablet 94abe840b1994c0cb1f7ac7f4ac09cd9
0501 13:57:48.348587 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:48.483250 (+134663us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:48.490050 (+ 6800us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:48.604171 (+114121us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1908057}
I20260501 13:57:48.605240 1421 leader_election.cc:304] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2
I20260501 13:57:48.605648 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 (table default.Table_8a2d4f67afd1427b8c32c5bb223a7148 [id=4668751f7bc649019547c38d0f8254c4]) successfully deleted
I20260501 13:57:48.595283 1882 raft_consensus.cc:359] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.606271 2720 raft_consensus.cc:2804] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:48.606320 1882 raft_consensus.cc:385] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.606604 1882 raft_consensus.cc:740] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:48.607616 2720 raft_consensus.cc:697] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Becoming Leader. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Running, Role: LEADER
I20260501 13:57:48.608386 2720 consensus_queue.cc:237] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.610647 2582 tablet_replica.cc:333] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:48.611435 2582 raft_consensus.cc:2243] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:48.611924 2582 raft_consensus.cc:2272] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:48.613515 2582 ts_tablet_manager.cc:1916] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:48.614519 1882 consensus_queue.cc:260] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.582430 2047 log.cc:1199] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/3eb6d30bed2a410db8b62fdff54a5fac
I20260501 13:57:48.616844 2047 ts_tablet_manager.cc:1950] T 3eb6d30bed2a410db8b62fdff54a5fac P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:48.620055 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_d564e785b85540ce834d43b55fda7ff0" } new_table_name: "default.Table_e7cc5f06279d4eb08c300dd9ad861bce" modify_external_catalogs: true
I20260501 13:57:48.621347 1882 ts_tablet_manager.cc:1434] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260501 13:57:48.622030 2726 catalog_manager.cc:1269] Loaded cluster ID: 5d0126731b4a4ae2a770b98ff0c83307
I20260501 13:57:48.622219 1882 tablet_bootstrap.cc:492] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:48.622295 2726 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: loading cluster ID for follower catalog manager: success
I20260501 13:57:48.646701 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 3eb6d30bed2a410db8b62fdff54a5fac (table default.Table_52acb3d1a0d1450bb41440f4de75ac91 [id=7b430735e8b640dc84cf92f308f20362]) successfully deleted
I20260501 13:57:48.648639 1881 tablet_bootstrap.cc:492] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:48.649591 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 396 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
I20260501 13:57:48.654036 1883 tablet_bootstrap.cc:492] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:48.654986 1883 ts_tablet_manager.cc:1403] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.128s user 0.008s sys 0.002s
I20260501 13:57:48.657682 1883 raft_consensus.cc:359] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.662585 1883 raft_consensus.cc:385] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.662915 1883 raft_consensus.cc:740] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:48.664644 2485 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" instance_seqno: 1777643846183296) as {username='slave'} at 127.1.16.131:50923; Asking this server to re-register.
I20260501 13:57:48.669409 1883 consensus_queue.cc:260] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:48.671960 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 678de81f2adb4ae6a2fce7fbfd1b08e7 has no consensus state
I20260501 13:57:48.667845 1828 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:48.682981 1882 tablet_bootstrap.cc:654] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:48.686504 1883 ts_tablet_manager.cc:1434] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.021s user 0.004s sys 0.001s
I20260501 13:57:48.683490 2047 tablet_replica.cc:333] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:48.687388 1883 tablet_bootstrap.cc:492] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:48.690739 2486 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" instance_seqno: 1777643844273168) as {username='slave'} at 127.1.16.130:39889; Asking this server to re-register.
I20260501 13:57:48.690146 1881 ts_tablet_manager.cc:1403] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.197s user 0.005s sys 0.005s
I20260501 13:57:48.693799 1675 heartbeater.cc:461] Registering TS with master...
I20260501 13:57:48.697822 1124 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_8a2d4f67afd1427b8c32c5bb223a7148" } new_table_name: "default.Table_404297cf87434f66977da0de3efdc121" modify_external_catalogs: true
I20260501 13:57:48.709357 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_0edc079d8a294a99aaca7a5847ae718d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:48.709892 2726 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: acquiring CA information for follower catalog manager: success
W20260501 13:57:48.711068 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0edc079d8a294a99aaca7a5847ae718d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.711855 2485 ts_manager.cc:194] Registered new tserver with Master: 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:48.713307 1881 raft_consensus.cc:359] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.720043 1883 tablet_bootstrap.cc:654] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:48.724957 2582 ts_tablet_manager.cc:1929] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:48.725332 2582 log.cc:1199] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/51ba7b8a7f4d4ec6ba4af775596d3b59
I20260501 13:57:48.725914 2582 ts_tablet_manager.cc:1950] T 51ba7b8a7f4d4ec6ba4af775596d3b59 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:48.733145 1882 tablet_bootstrap.cc:492] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:48.733557 1882 ts_tablet_manager.cc:1403] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.111s user 0.009s sys 0.000s
I20260501 13:57:48.735849 1882 raft_consensus.cc:359] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.736472 1882 raft_consensus.cc:385] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.736730 1882 raft_consensus.cc:740] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:48.738781 1215 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 396 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" is_pre_election: true
I20260501 13:57:48.739384 1882 consensus_queue.cc:260] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.740597 2451 leader_election.cc:304] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50c0932256e74399a537670bb424a1b7; no voters: d18ed9dc598e4ccaaf8633eaa45a2f51, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:48.741834 2635 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260501 13:57:48.745728 1124 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74af56cca54347bab67895e10ee17230 consensus state has no leader
I20260501 13:57:48.715108 2486 ts_manager.cc:194] Registered new tserver with Master: 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:48.750674 1882 ts_tablet_manager.cc:1434] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260501 13:57:48.731510 2047 raft_consensus.cc:2243] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:48.754781 1882 tablet_bootstrap.cc:492] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:48.755007 2047 pending_rounds.cc:70] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: Trying to abort 1 pending ops.
I20260501 13:57:48.755223 2047 pending_rounds.cc:77] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229285844791296 op_type: NO_OP noop_request { }
I20260501 13:57:48.755522 2047 raft_consensus.cc:2889] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:57:48.755836 2047 raft_consensus.cc:2272] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:48.756352 1881 raft_consensus.cc:385] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.756757 1881 raft_consensus.cc:740] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:48.757560 1881 consensus_queue.cc:260] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.777673 2047 ts_tablet_manager.cc:1916] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:48.778570 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 317) took 2357 ms. Trace:
I20260501 13:57:48.778921 2582 rpcz_store.cc:276] 0501 13:57:46.421056 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.421482 (+ 426us) service_pool.cc:224] Handling call
0501 13:57:48.610490 (+2189008us) ts_tablet_manager.cc:1151] Deleting tablet 51ba7b8a7f4d4ec6ba4af775596d3b59
0501 13:57:48.610531 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:48.685285 (+ 74754us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:48.724822 (+ 39537us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:48.778506 (+ 53684us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2169738}
I20260501 13:57:48.780067 1464 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c736959df334aee857dbaae0f7a1e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:40 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:48.780733 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 51ba7b8a7f4d4ec6ba4af775596d3b59 (table default.Table_7b8be07c742f48d0af3422dfea143dd1 [id=39aeda29815d474196136813b5d811e7]) successfully deleted
I20260501 13:57:48.735574 2726 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 13:57:48.796120 2644 raft_consensus.cc:493] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:48.796587 2644 raft_consensus.cc:515] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.802778 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "569a0240b1274d9bb830addbaf1ec343" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:48.804482 1564 leader_election.cc:343] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:48.805434 1563 leader_election.cc:343] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 569a0240b1274d9bb830addbaf1ec343
I20260501 13:57:48.805755 1563 leader_election.cc:304] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:48.806540 2685 raft_consensus.cc:2749] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:48.798364 2644 leader_election.cc:290] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:48.809491 1883 tablet_bootstrap.cc:492] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:48.809979 1883 ts_tablet_manager.cc:1403] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.123s user 0.002s sys 0.009s
I20260501 13:57:48.796482 1881 ts_tablet_manager.cc:1434] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.103s user 0.003s sys 0.002s
I20260501 13:57:48.821807 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_5b3bd4a6c9954c61bf60fbdffecbe586"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:48.822921 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 321) took 2333 ms. Trace:
I20260501 13:57:48.823251 2582 rpcz_store.cc:276] 0501 13:57:46.489838 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.489975 (+ 137us) service_pool.cc:224] Handling call
0501 13:57:48.822595 (+2332620us) ts_tablet_manager.cc:1151] Deleting tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd
0501 13:57:48.822640 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:48.822867 (+ 227us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2332051}
I20260501 13:57:48.825753 1883 raft_consensus.cc:359] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.827137 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 322) took 2337 ms. Trace:
I20260501 13:57:48.827419 2582 rpcz_store.cc:276] 0501 13:57:46.490115 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.490206 (+ 91us) service_pool.cc:224] Handling call
0501 13:57:48.826830 (+2336624us) ts_tablet_manager.cc:1151] Deleting tablet 30a1ef8502c648ce8da3905e4dff4c0c
0501 13:57:48.826876 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:48.827088 (+ 212us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2336112}
I20260501 13:57:48.826354 1883 raft_consensus.cc:385] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:48.828415 1883 raft_consensus.cc:740] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
W20260501 13:57:48.832446 1564 consensus_peers.cc:597] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:48.835106 1881 tablet_bootstrap.cc:492] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
W20260501 13:57:48.824296 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd already in progress: creating tablet
W20260501 13:57:48.839802 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 30a1ef8502c648ce8da3905e4dff4c0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30a1ef8502c648ce8da3905e4dff4c0c already in progress: creating tablet
I20260501 13:57:48.828136 1124 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4b58b0177375487d90d0ce607bd08b04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:48.840548 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 3f1ec9cbfd5847d1832d7ea8d002baab (DEFAULT_TABLE table=default.Table_3b1e200ad2674105a1911b12abed2310 [id=c4f5ad2dafa64627aaee0449c4a4be2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.841517 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1ec9cbfd5847d1832d7ea8d002baab. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:48.841796 1124 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b58b0177375487d90d0ce607bd08b04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.843572 1602 tablet_service.cc:1511] Processing CreateTablet for tablet b076b2f04c88469ab5c3a23cd0c3c1a1 (DEFAULT_TABLE table=default.Table_ea2a86f75e984168b2aeb1606559389d [id=c443ca9457d647079d795b3089df1215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.844489 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b076b2f04c88469ab5c3a23cd0c3c1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.846668 1754 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ab056da506f454984a07026341c534b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:40 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:48.836018 1464 tablet_service.cc:1511] Processing CreateTablet for tablet 3f1ec9cbfd5847d1832d7ea8d002baab (DEFAULT_TABLE table=default.Table_3b1e200ad2674105a1911b12abed2310 [id=c4f5ad2dafa64627aaee0449c4a4be2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.838399 1456 tablet_service.cc:1511] Processing CreateTablet for tablet b076b2f04c88469ab5c3a23cd0c3c1a1 (DEFAULT_TABLE table=default.Table_ea2a86f75e984168b2aeb1606559389d [id=c443ca9457d647079d795b3089df1215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.847862 1761 tablet_service.cc:1511] Processing CreateTablet for tablet b076b2f04c88469ab5c3a23cd0c3c1a1 (DEFAULT_TABLE table=default.Table_ea2a86f75e984168b2aeb1606559389d [id=c443ca9457d647079d795b3089df1215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.848592 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1ec9cbfd5847d1832d7ea8d002baab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.848760 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b076b2f04c88469ab5c3a23cd0c3c1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.849126 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b076b2f04c88469ab5c3a23cd0c3c1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.850875 1608 tablet_service.cc:1511] Processing CreateTablet for tablet 7100bdb796c444759d7b191246739d6b (DEFAULT_TABLE table=default.Table_ce82bd9de8db4f84bb12d673fc45e30f [id=a2af904635984ca6951f4b393397d4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.850968 1609 tablet_service.cc:1511] Processing CreateTablet for tablet 8456dfee265d4030a145456a3c6bf05c (DEFAULT_TABLE table=default.Table_b62edaca3f654cd6a4dcd07badd05637 [id=acb8d18d86454c26bddcae1f83168538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.851857 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7100bdb796c444759d7b191246739d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.851931 1601 tablet_service.cc:1511] Processing CreateTablet for tablet febbcd4223bb468bb041f2ac01bac408 (DEFAULT_TABLE table=default.Table_aac3e9446b9e433a9cd6c8a08f7b7660 [id=1d6b92f98114446ebf52b71fde19053e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.852785 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet febbcd4223bb468bb041f2ac01bac408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.855194 1762 tablet_service.cc:1511] Processing CreateTablet for tablet febbcd4223bb468bb041f2ac01bac408 (DEFAULT_TABLE table=default.Table_aac3e9446b9e433a9cd6c8a08f7b7660 [id=1d6b92f98114446ebf52b71fde19053e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.856163 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet febbcd4223bb468bb041f2ac01bac408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.847868 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 3f1ec9cbfd5847d1832d7ea8d002baab (DEFAULT_TABLE table=default.Table_3b1e200ad2674105a1911b12abed2310 [id=c4f5ad2dafa64627aaee0449c4a4be2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.859791 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1ec9cbfd5847d1832d7ea8d002baab. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:48.861343 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b3bd4a6c9954c61bf60fbdffecbe586 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.862957 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8456dfee265d4030a145456a3c6bf05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.871402 1610 tablet_service.cc:1511] Processing CreateTablet for tablet eed21cf78046471481c81b3157177910 (DEFAULT_TABLE table=default.Table_5bbee1238b46474e808a9fca817eefa0 [id=4523b625c7104ebca43d4ff44922687a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.872367 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed21cf78046471481c81b3157177910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.877574 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 7100bdb796c444759d7b191246739d6b (DEFAULT_TABLE table=default.Table_ce82bd9de8db4f84bb12d673fc45e30f [id=a2af904635984ca6951f4b393397d4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.878082 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 8456dfee265d4030a145456a3c6bf05c (DEFAULT_TABLE table=default.Table_b62edaca3f654cd6a4dcd07badd05637 [id=acb8d18d86454c26bddcae1f83168538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.878566 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7100bdb796c444759d7b191246739d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.879443 1757 tablet_service.cc:1511] Processing CreateTablet for tablet eed21cf78046471481c81b3157177910 (DEFAULT_TABLE table=default.Table_5bbee1238b46474e808a9fca817eefa0 [id=4523b625c7104ebca43d4ff44922687a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.880362 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed21cf78046471481c81b3157177910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.883122 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 54caa456f05e478a839235a16c17eda1 (DEFAULT_TABLE table=default.Table_3d46da834ff64e5b9b54a0a28295ed4b [id=04f11bb825dc49e28136cf64ff9f3469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.884039 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54caa456f05e478a839235a16c17eda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.886730 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8456dfee265d4030a145456a3c6bf05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.887092 1607 tablet_service.cc:1511] Processing CreateTablet for tablet 54caa456f05e478a839235a16c17eda1 (DEFAULT_TABLE table=default.Table_3d46da834ff64e5b9b54a0a28295ed4b [id=04f11bb825dc49e28136cf64ff9f3469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.887985 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54caa456f05e478a839235a16c17eda1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:48.832433 1125 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.131:42895 (request call id 78) took 1274 ms (client timeout 1000 ms). Trace:
W20260501 13:57:48.890563 1125 rpcz_store.cc:269] 0501 13:57:47.557781 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.557891 (+ 110us) service_pool.cc:224] Handling call
0501 13:57:48.832368 (+1274477us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":2}
I20260501 13:57:48.893289 2734 raft_consensus.cc:493] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 13:57:48.836880 1132 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.130:35289 (request call id 78) took 1438 ms (client timeout 1000 ms). Trace:
W20260501 13:57:48.893736 1132 rpcz_store.cc:269] 0501 13:57:47.398716 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.398813 (+ 97us) service_pool.cc:224] Handling call
0501 13:57:48.836815 (+1438002us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260501 13:57:48.893708 2734 raft_consensus.cc:515] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.895501 2734 leader_election.cc:290] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:48.837859 1123 catalog_manager.cc:5671] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:48.896446 1123 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 56cee07054a64e199a0bfc26b63eca2a (table default.Table_9b824b59406d478884838d29a5d690f8 [id=389132d3db474d008cc0ee9d758b4fec]). Expected version 1 got 0
I20260501 13:57:48.897030 1123 catalog_manager.cc:5671] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:48.898168 1123 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 7d399de014d74cab8df14f44787ca2f7 (table default.Table_aa47824eda84455d9d52df61514a253e [id=81117c514ffd4ab6a8d4dde20d7559e4]). Expected version 1 got 0
I20260501 13:57:48.898630 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dbb619863dd405ab20ead6bbebf4d69" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:48.900002 1719 leader_election.cc:343] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:48.901834 1753 tablet_service.cc:1511] Processing CreateTablet for tablet f26a3d15d6a54cfa98e2504b54200e0c (DEFAULT_TABLE table=default.Table_fe1f71fadf9546cd98a2f81acc82bce6 [id=30ed2ee323d54634935ebe65a22ddc31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.903059 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f26a3d15d6a54cfa98e2504b54200e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.911474 1755 tablet_service.cc:1511] Processing CreateTablet for tablet dca682bccce84ced8dc90b9519ec5230 (DEFAULT_TABLE table=default.Table_6d06a2f21f3140abb7a05eeee1b563ae [id=ecaaae3a740f4bf48d27452dcae09935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.912490 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dca682bccce84ced8dc90b9519ec5230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.916452 1123 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:48.919596 1131 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 9d38d75fcd494fc4a3420bcf15f2c70f
I20260501 13:57:48.920807 1131 catalog_manager.cc:6741] ReplaceTablet: tablet 9d38d75fcd494fc4a3420bcf15f2c70f deleted and replaced by tablet 9f677dcbf0f540749ff71906240b703b
I20260501 13:57:48.922349 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_305c654ad4204e5d93626ca91d5731ce" } new_table_name: "default.Table_6b367195dbab461e9a822d9ee5facdc1" modify_external_catalogs: true
I20260501 13:57:48.918211 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_4ce0c9af4af54afbb1b2bb23a5edd136"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:48.945333 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ce0c9af4af54afbb1b2bb23a5edd136 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:48.950001 1606 tablet_service.cc:1511] Processing CreateTablet for tablet f26a3d15d6a54cfa98e2504b54200e0c (DEFAULT_TABLE table=default.Table_fe1f71fadf9546cd98a2f81acc82bce6 [id=30ed2ee323d54634935ebe65a22ddc31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.950227 1604 tablet_service.cc:1511] Processing CreateTablet for tablet dca682bccce84ced8dc90b9519ec5230 (DEFAULT_TABLE table=default.Table_6d06a2f21f3140abb7a05eeee1b563ae [id=ecaaae3a740f4bf48d27452dcae09935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.951324 1603 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d38d75fcd494fc4a3420bcf15f2c70f with delete_type TABLET_DATA_DELETED (replaced by tablet 9f677dcbf0f540749ff71906240b703b) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:48.952489 1600 tablet_service.cc:1511] Processing CreateTablet for tablet d9649a8cb0e641fcbbee1bd080a23e0b (DEFAULT_TABLE table=default.Table_dc76e09f85ed44ff96e6f2f1a7a1c223 [id=1870d700728c40ba88f0f0a015fb8a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.961320 1883 consensus_queue.cc:260] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.963181 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f26a3d15d6a54cfa98e2504b54200e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.964905 1462 tablet_service.cc:1511] Processing CreateTablet for tablet febbcd4223bb468bb041f2ac01bac408 (DEFAULT_TABLE table=default.Table_aac3e9446b9e433a9cd6c8a08f7b7660 [id=1d6b92f98114446ebf52b71fde19053e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.965874 1462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet febbcd4223bb468bb041f2ac01bac408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.967043 1600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9649a8cb0e641fcbbee1bd080a23e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.970965 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dca682bccce84ced8dc90b9519ec5230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.973203 1752 tablet_service.cc:1511] Processing CreateTablet for tablet d9649a8cb0e641fcbbee1bd080a23e0b (DEFAULT_TABLE table=default.Table_dc76e09f85ed44ff96e6f2f1a7a1c223 [id=1870d700728c40ba88f0f0a015fb8a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.974193 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9649a8cb0e641fcbbee1bd080a23e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.979666 1758 tablet_service.cc:1511] Processing CreateTablet for tablet 335b39f284394c51a22a6aeee99abddc (DEFAULT_TABLE table=default.Table_51d3bb23841a45c78ff5cc7e492e0c14 [id=0b822c8a4c25486d84b8d1eaac6d0c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.980695 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 335b39f284394c51a22a6aeee99abddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.983687 1751 tablet_service.cc:1511] Processing CreateTablet for tablet 678de81f2adb4ae6a2fce7fbfd1b08e7 (DEFAULT_TABLE table=default.Table_e7cc5f06279d4eb08c300dd9ad861bce [id=bbb98b9cca114562b0cefa20c58ac794]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.984742 1751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678de81f2adb4ae6a2fce7fbfd1b08e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.988951 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 323) took 2498 ms. Trace:
I20260501 13:57:48.963794 2719 raft_consensus.cc:493] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:48.992964 2719 raft_consensus.cc:515] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:48.994827 2719 leader_election.cc:290] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:48.996892 1465 tablet_service.cc:1511] Processing CreateTablet for tablet eed21cf78046471481c81b3157177910 (DEFAULT_TABLE table=default.Table_5bbee1238b46474e808a9fca817eefa0 [id=4523b625c7104ebca43d4ff44922687a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.997179 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 54caa456f05e478a839235a16c17eda1 (DEFAULT_TABLE table=default.Table_3d46da834ff64e5b9b54a0a28295ed4b [id=04f11bb825dc49e28136cf64ff9f3469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.997812 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed21cf78046471481c81b3157177910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.998850 1455 tablet_service.cc:1511] Processing CreateTablet for tablet 7100bdb796c444759d7b191246739d6b (DEFAULT_TABLE table=default.Table_ce82bd9de8db4f84bb12d673fc45e30f [id=a2af904635984ca6951f4b393397d4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.999301 1454 tablet_service.cc:1511] Processing CreateTablet for tablet 678de81f2adb4ae6a2fce7fbfd1b08e7 (DEFAULT_TABLE table=default.Table_e7cc5f06279d4eb08c300dd9ad861bce [id=bbb98b9cca114562b0cefa20c58ac794]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:48.999714 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7100bdb796c444759d7b191246739d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.000281 1454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678de81f2adb4ae6a2fce7fbfd1b08e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.000924 1458 tablet_service.cc:1511] Processing CreateTablet for tablet f26a3d15d6a54cfa98e2504b54200e0c (DEFAULT_TABLE table=default.Table_fe1f71fadf9546cd98a2f81acc82bce6 [id=30ed2ee323d54634935ebe65a22ddc31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.001724 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f26a3d15d6a54cfa98e2504b54200e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.003372 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d38d75fcd494fc4a3420bcf15f2c70f with delete_type TABLET_DATA_DELETED (replaced by tablet 9f677dcbf0f540749ff71906240b703b) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:49.003639 1418 leader_election.cc:343] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 94abe840b1994c0cb1f7ac7f4ac09cd9
I20260501 13:57:48.998389 1463 tablet_service.cc:1511] Processing CreateTablet for tablet 335b39f284394c51a22a6aeee99abddc (DEFAULT_TABLE table=default.Table_51d3bb23841a45c78ff5cc7e492e0c14 [id=0b822c8a4c25486d84b8d1eaac6d0c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.006142 1460 tablet_service.cc:1511] Processing CreateTablet for tablet dca682bccce84ced8dc90b9519ec5230 (DEFAULT_TABLE table=default.Table_6d06a2f21f3140abb7a05eeee1b563ae [id=ecaaae3a740f4bf48d27452dcae09935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.006817 1463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 335b39f284394c51a22a6aeee99abddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.007527 1457 tablet_service.cc:1511] Processing CreateTablet for tablet d9649a8cb0e641fcbbee1bd080a23e0b (DEFAULT_TABLE table=default.Table_dc76e09f85ed44ff96e6f2f1a7a1c223 [id=1870d700728c40ba88f0f0a015fb8a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.008448 1457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9649a8cb0e641fcbbee1bd080a23e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.010710 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54caa456f05e478a839235a16c17eda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.010669 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d38d75fcd494fc4a3420bcf15f2c70f with delete_type TABLET_DATA_DELETED (replaced by tablet 9f677dcbf0f540749ff71906240b703b) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:49.014328 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dbb619863dd405ab20ead6bbebf4d69" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:49.016000 1717 leader_election.cc:343] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:49.016395 1717 leader_election.cc:304] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:49.024151 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dca682bccce84ced8dc90b9519ec5230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.018380 1603 tablet_service.cc:1511] Processing CreateTablet for tablet 335b39f284394c51a22a6aeee99abddc (DEFAULT_TABLE table=default.Table_51d3bb23841a45c78ff5cc7e492e0c14 [id=0b822c8a4c25486d84b8d1eaac6d0c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.029268 1750 tablet_service.cc:1511] Processing CreateTablet for tablet 6c230ae9a84d466b8525acde315084e3 (DEFAULT_TABLE table=default.Table_769d5d15b713487f9183c5bc67155757 [id=a61b1d9a32654f03a510d3cb3bf32068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.029768 1603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 335b39f284394c51a22a6aeee99abddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.030436 1750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c230ae9a84d466b8525acde315084e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.020145 1599 tablet_service.cc:1511] Processing CreateTablet for tablet 678de81f2adb4ae6a2fce7fbfd1b08e7 (DEFAULT_TABLE table=default.Table_e7cc5f06279d4eb08c300dd9ad861bce [id=bbb98b9cca114562b0cefa20c58ac794]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.021344 1598 tablet_service.cc:1511] Processing CreateTablet for tablet 6c230ae9a84d466b8525acde315084e3 (DEFAULT_TABLE table=default.Table_769d5d15b713487f9183c5bc67155757 [id=a61b1d9a32654f03a510d3cb3bf32068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.035270 1599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678de81f2adb4ae6a2fce7fbfd1b08e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.022485 1597 tablet_service.cc:1511] Processing CreateTablet for tablet a3801a20a11543bdb2cb9a5961308c71 (DEFAULT_TABLE table=default.Table_ecb1d6e04f584898ae14b3aec89681a2 [id=dfea1dc1796341b99f55135d7e4f4d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.036145 1597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3801a20a11543bdb2cb9a5961308c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.038852 1598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c230ae9a84d466b8525acde315084e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.036262 2582 rpcz_store.cc:276] 0501 13:57:46.490379 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.491241 (+ 862us) service_pool.cc:224] Handling call
0501 13:57:48.950064 (+2458823us) ts_tablet_manager.cc:1151] Deleting tablet 6c7e65dad084440787754e24cdd579d9
0501 13:57:48.950107 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:48.988887 (+ 38780us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2458313}
I20260501 13:57:49.042129 1478 raft_consensus.cc:3060] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:49.042083 1453 tablet_service.cc:1511] Processing CreateTablet for tablet a3801a20a11543bdb2cb9a5961308c71 (DEFAULT_TABLE table=default.Table_ecb1d6e04f584898ae14b3aec89681a2 [id=dfea1dc1796341b99f55135d7e4f4d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.042699 1759 tablet_service.cc:1511] Processing CreateTablet for tablet a3801a20a11543bdb2cb9a5961308c71 (DEFAULT_TABLE table=default.Table_ecb1d6e04f584898ae14b3aec89681a2 [id=dfea1dc1796341b99f55135d7e4f4d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.043249 1453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3801a20a11543bdb2cb9a5961308c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.044786 1459 tablet_service.cc:1511] Processing CreateTablet for tablet 6c230ae9a84d466b8525acde315084e3 (DEFAULT_TABLE table=default.Table_769d5d15b713487f9183c5bc67155757 [id=a61b1d9a32654f03a510d3cb3bf32068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.045691 1459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c230ae9a84d466b8525acde315084e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:49.047461 1421 leader_election.cc:343] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Not found: Tablet not found: 94abe840b1994c0cb1f7ac7f4ac09cd9
I20260501 13:57:49.047832 1421 leader_election.cc:304] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
I20260501 13:57:49.050678 2720 raft_consensus.cc:2749] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:49.051504 1563 consensus_peers.cc:597] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Couldn't send request to peer 0ed1f9a545474e76a661c32bf11a7fb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3eb6d30bed2a410db8b62fdff54a5fac. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:49.053124 1127 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet bbb10f2c3d864931abb3ae2a88372d61
I20260501 13:57:49.054519 1127 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41700 (request call id 445) took 1350 ms. Trace:
I20260501 13:57:49.055590 1452 tablet_service.cc:1511] Processing CreateTablet for tablet 36b9a9c6e5c340f68b071c9a9ae8d464 (DEFAULT_TABLE table=default.Table_31b7fbe1002b47b0adb3e3abb1517807 [id=e2c1080f71804f5b8f68ac9fbcfeb7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.056756 1452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b9a9c6e5c340f68b071c9a9ae8d464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.054862 1127 rpcz_store.cc:276] 0501 13:57:47.703988 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.712126 (+ 8138us) service_pool.cc:224] Handling call
0501 13:57:47.719503 (+ 7377us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:57:47.720886 (+ 1383us) catalog_manager.cc:2815] Modifying in-memory table state
0501 13:57:47.721012 (+ 126us) catalog_manager.cc:2824] Locking tablets
0501 13:57:48.840214 (+1119202us) catalog_manager.cc:2838] Removing table and tablets from system table
0501 13:57:49.052780 (+212566us) catalog_manager.cc:2856] Removing table from by-name map
0501 13:57:49.052966 (+ 186us) catalog_manager.cc:2872] Committing in-memory state
0501 13:57:49.053046 (+ 80us) catalog_manager.cc:2876] Aborting table tasks
0501 13:57:49.054414 (+ 1368us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:48.868818 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:48.869166 (+ 348us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:48.869188 (+ 22us) write_op.cc:435] Acquired schema lock
0501 13:57:48.869200 (+ 12us) tablet.cc:662] Decoding operations
0501 13:57:48.869474 (+ 274us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:48.869508 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:48.869529 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:57:48.869765 (+ 236us) tablet.cc:701] Row locks acquired
0501 13:57:48.869778 (+ 13us) write_op.cc:260] PREPARE: finished
0501 13:57:48.869880 (+ 102us) write_op.cc:270] Start()
0501 13:57:48.869952 (+ 72us) write_op.cc:276] Timestamp: P: 1777643868869884 usec, L: 0
0501 13:57:48.869971 (+ 19us) op_driver.cc:348] REPLICATION: starting
0501 13:57:48.870412 (+ 441us) log.cc:844] Serialized 728 byte log entry
0501 13:57:49.049062 (+178650us) op_driver.cc:464] REPLICATION: finished
0501 13:57:49.050635 (+ 1573us) write_op.cc:301] APPLY: starting
0501 13:57:49.050701 (+ 66us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:49.050807 (+ 106us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:49.050899 (+ 92us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:49.051138 (+ 239us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:49.051277 (+ 139us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0501 13:57:49.051322 (+ 45us) write_op.cc:312] APPLY: finished
0501 13:57:49.051506 (+ 184us) log.cc:844] Serialized 29 byte log entry
0501 13:57:49.051667 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:49.051735 (+ 68us) write_op.cc:454] Released schema lock
0501 13:57:49.051765 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:49.051991 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:49.052016 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":1957248,"child_traces":[["op",{"apply.queue_time_us":1396,"apply.run_cpu_time_us":1327,"apply.run_wall_time_us":1327,"num_ops":2,"prepare.queue_time_us":27051,"prepare.run_cpu_time_us":1807,"prepare.run_wall_time_us":1808,"replication_time_us":178996}]]}
I20260501 13:57:49.059873 1127 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_cf75782d380e49e3b4cf46f49c816974" } new_table_name: "default.Table_6ae250fd6cc242ba815b6d5ace5aafbb" modify_external_catalogs: true
I20260501 13:57:49.071951 1451 tablet_service.cc:1511] Processing CreateTablet for tablet 0fdf7c660b624c86ad873f8a124f5f21 (DEFAULT_TABLE table=default.Table_cdc592203f384ccd8c0142007f14d5ff [id=d1627e3ff79448c38e8cde54c4981219]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.073087 1451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdf7c660b624c86ad873f8a124f5f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.072858 1749 tablet_service.cc:1511] Processing CreateTablet for tablet 36b9a9c6e5c340f68b071c9a9ae8d464 (DEFAULT_TABLE table=default.Table_31b7fbe1002b47b0adb3e3abb1517807 [id=e2c1080f71804f5b8f68ac9fbcfeb7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.075245 1748 tablet_service.cc:1511] Processing CreateTablet for tablet 0fdf7c660b624c86ad873f8a124f5f21 (DEFAULT_TABLE table=default.Table_cdc592203f384ccd8c0142007f14d5ff [id=d1627e3ff79448c38e8cde54c4981219]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.076936 1747 tablet_service.cc:1511] Processing CreateTablet for tablet 141738e2b87743b88492d66b2cfff2d8 (DEFAULT_TABLE table=default.Table_1a37845ff6b34f87ae7b12cbc4738328 [id=7d409e16592a4782af565f7ff24fe10c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.078635 1746 tablet_service.cc:1511] Processing CreateTablet for tablet da698cb245ee42e6bd02e7b342035372 (DEFAULT_TABLE table=default.Table_aab8dd1a3e894677b333dc115aa2222c [id=5c6280db5c9d446f87ddf965c517583f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.079798 1450 tablet_service.cc:1511] Processing CreateTablet for tablet 141738e2b87743b88492d66b2cfff2d8 (DEFAULT_TABLE table=default.Table_1a37845ff6b34f87ae7b12cbc4738328 [id=7d409e16592a4782af565f7ff24fe10c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.080286 1745 tablet_service.cc:1511] Processing CreateTablet for tablet 862b53262b134023ac7f990f3d3fe734 (DEFAULT_TABLE table=default.Table_4e3867de9f9848d99ecb5f0dd886d117 [id=f37873170cf1460b918bd07438190302]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.080900 1450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141738e2b87743b88492d66b2cfff2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.081954 1744 tablet_service.cc:1511] Processing CreateTablet for tablet 6f9f47b997b14d3a840044a4cf74fa10 (DEFAULT_TABLE table=default.Table_1f1a1e3878d641fc88bcb55193ac5d67 [id=1edfc147d6b341c589d211078c2bf585]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.083756 1449 tablet_service.cc:1511] Processing CreateTablet for tablet da698cb245ee42e6bd02e7b342035372 (DEFAULT_TABLE table=default.Table_aab8dd1a3e894677b333dc115aa2222c [id=5c6280db5c9d446f87ddf965c517583f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.084867 1449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da698cb245ee42e6bd02e7b342035372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.088271 1448 tablet_service.cc:1511] Processing CreateTablet for tablet 862b53262b134023ac7f990f3d3fe734 (DEFAULT_TABLE table=default.Table_4e3867de9f9848d99ecb5f0dd886d117 [id=f37873170cf1460b918bd07438190302]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.090943 1448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862b53262b134023ac7f990f3d3fe734. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:49.091900 1117 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.189:45897, user_credentials={real_user=slave}} blocked reactor thread for 67317.3us
I20260501 13:57:49.093930 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 8456dfee265d4030a145456a3c6bf05c (DEFAULT_TABLE table=default.Table_b62edaca3f654cd6a4dcd07badd05637 [id=acb8d18d86454c26bddcae1f83168538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.095037 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8456dfee265d4030a145456a3c6bf05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:48.765780 1882 tablet_bootstrap.cc:654] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:49.105324 1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 89368.6us
I20260501 13:57:49.105948 2734 raft_consensus.cc:2749] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:49.107690 1883 ts_tablet_manager.cc:1434] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.297s user 0.003s sys 0.002s
I20260501 13:57:49.108601 1883 tablet_bootstrap.cc:492] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:49.133145 1749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b9a9c6e5c340f68b071c9a9ae8d464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.134662 1747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141738e2b87743b88492d66b2cfff2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.135986 1745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862b53262b134023ac7f990f3d3fe734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.137149 1746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da698cb245ee42e6bd02e7b342035372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.138311 1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f9f47b997b14d3a840044a4cf74fa10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.139531 1748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdf7c660b624c86ad873f8a124f5f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.141414 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3801a20a11543bdb2cb9a5961308c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.148487 2734 raft_consensus.cc:493] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:49.148923 2734 raft_consensus.cc:515] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.150581 2734 leader_election.cc:290] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:49.170572 2047 ts_tablet_manager.cc:1929] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:49.189385 2047 log.cc:1199] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/38c3eb388d474d718554775d65f58d78
I20260501 13:57:49.190228 2047 ts_tablet_manager.cc:1950] T 38c3eb388d474d718554775d65f58d78 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:49.191063 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 332) took 1466 ms. Trace:
I20260501 13:57:49.191465 2047 rpcz_store.cc:276] 0501 13:57:47.725162 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.725266 (+ 104us) service_pool.cc:224] Handling call
0501 13:57:48.681738 (+956472us) ts_tablet_manager.cc:1151] Deleting tablet 38c3eb388d474d718554775d65f58d78
0501 13:57:48.682741 (+ 1003us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:48.812802 (+130061us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:49.170428 (+357626us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:49.191005 (+ 20577us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":955799}
I20260501 13:57:49.192734 2047 tablet_replica.cc:333] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:49.139591 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 6ccedb53d6f84c0a96766fae070f191e (DEFAULT_TABLE table=default.Table_3b2f8dac784c40d0ae02c8d1fcd3a51e [id=0581ada3258946a3b3bb9a2a6d15b3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.228003 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ccedb53d6f84c0a96766fae070f191e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.229171 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 325) took 2611 ms. Trace:
I20260501 13:57:49.229516 2582 rpcz_store.cc:276] 0501 13:57:46.618022 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.618718 (+ 696us) service_pool.cc:224] Handling call
0501 13:57:49.039935 (+2421217us) ts_tablet_manager.cc:1151] Deleting tablet a000b129a6ab429889b543c20750a493
0501 13:57:49.039977 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:49.229110 (+189133us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2420719}
I20260501 13:57:49.230525 2582 tablet_replica.cc:333] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:49.237195 2582 raft_consensus.cc:2243] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:49.254858 1595 tablet_service.cc:1511] Processing CreateTablet for tablet 0fdf7c660b624c86ad873f8a124f5f21 (DEFAULT_TABLE table=default.Table_cdc592203f384ccd8c0142007f14d5ff [id=d1627e3ff79448c38e8cde54c4981219]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.256654 2734 raft_consensus.cc:493] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:49.257162 2734 raft_consensus.cc:515] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.258486 2582 raft_consensus.cc:2272] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:49.264268 1595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdf7c660b624c86ad873f8a124f5f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.208465 1447 tablet_service.cc:1511] Processing CreateTablet for tablet 6f9f47b997b14d3a840044a4cf74fa10 (DEFAULT_TABLE table=default.Table_1f1a1e3878d641fc88bcb55193ac5d67 [id=1edfc147d6b341c589d211078c2bf585]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.265187 2047 raft_consensus.cc:2243] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:49.214526 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "967658b1be204c638fc34983293ef5ab" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:49.276803 2644 consensus_queue.cc:1048] T 9169851f1358460d99a48cb6067ab3e3 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:57:49.268332 1596 tablet_service.cc:1511] Processing CreateTablet for tablet 36b9a9c6e5c340f68b071c9a9ae8d464 (DEFAULT_TABLE table=default.Table_31b7fbe1002b47b0adb3e3abb1517807 [id=e2c1080f71804f5b8f68ac9fbcfeb7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.280284 1596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b9a9c6e5c340f68b071c9a9ae8d464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.267686 2685 raft_consensus.cc:493] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:49.281487 2685 raft_consensus.cc:515] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.284053 1594 tablet_service.cc:1511] Processing CreateTablet for tablet 141738e2b87743b88492d66b2cfff2d8 (DEFAULT_TABLE table=default.Table_1a37845ff6b34f87ae7b12cbc4738328 [id=7d409e16592a4782af565f7ff24fe10c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.285200 1594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141738e2b87743b88492d66b2cfff2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.295305 2685 leader_election.cc:290] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:49.296590 2582 ts_tablet_manager.cc:1916] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:49.304386 1601 tablet_service.cc:1511] Processing CreateTablet for tablet 6ccedb53d6f84c0a96766fae070f191e (DEFAULT_TABLE table=default.Table_3b2f8dac784c40d0ae02c8d1fcd3a51e [id=0581ada3258946a3b3bb9a2a6d15b3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.221583 1460 tablet_service.cc:1511] Processing CreateTablet for tablet 6ccedb53d6f84c0a96766fae070f191e (DEFAULT_TABLE table=default.Table_3b2f8dac784c40d0ae02c8d1fcd3a51e [id=0581ada3258946a3b3bb9a2a6d15b3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.233520 1458 tablet_service.cc:1511] Processing CreateTablet for tablet b977eff59197438ea161d25742031723 (DEFAULT_TABLE table=default.Table_62d1e55cf6304e208c648b8d70b41550 [id=004c240f66f44847a97ce78efa3d3cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.305184 1593 tablet_service.cc:1511] Processing CreateTablet for tablet 2ad83fbc8f544e7b9813774368dbb062 (DEFAULT_TABLE table=default.Table_fdbac3c4239747e396088091988a1794 [id=5585064fd41841beb6a9ccc2e30ea5af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.307670 1592 tablet_service.cc:1511] Processing CreateTablet for tablet e7db988380fe4860881532072b10eefc (DEFAULT_TABLE table=default.Table_33fb06bbd21c4209919e7b4711c86e48 [id=49870caf0d0d49059424859f3a594352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.311233 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b977eff59197438ea161d25742031723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.309835 1591 tablet_service.cc:1511] Processing CreateTablet for tablet e527fe78682b4a319c8890ba52800099 (DEFAULT_TABLE table=default.Table_96a8aa986c4a42bcbc6e2655bd886c5d [id=8cd7315717a74b7d9bf8f54ef46768f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.311887 1593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad83fbc8f544e7b9813774368dbb062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.312847 1756 tablet_service.cc:1511] Processing CreateTablet for tablet 2ad83fbc8f544e7b9813774368dbb062 (DEFAULT_TABLE table=default.Table_fdbac3c4239747e396088091988a1794 [id=5585064fd41841beb6a9ccc2e30ea5af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.313791 1756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad83fbc8f544e7b9813774368dbb062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.235839 1881 tablet_bootstrap.cc:654] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:49.315394 1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e527fe78682b4a319c8890ba52800099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.315908 1460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ccedb53d6f84c0a96766fae070f191e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.317538 1592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7db988380fe4860881532072b10eefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.323837 1605 tablet_service.cc:1511] Processing CreateTablet for tablet 6f9f47b997b14d3a840044a4cf74fa10 (DEFAULT_TABLE table=default.Table_1f1a1e3878d641fc88bcb55193ac5d67 [id=1edfc147d6b341c589d211078c2bf585]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.324775 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f9f47b997b14d3a840044a4cf74fa10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.334925 2047 raft_consensus.cc:2272] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:49.336793 2047 ts_tablet_manager.cc:1916] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:49.339624 1447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f9f47b997b14d3a840044a4cf74fa10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.310691 1608 tablet_service.cc:1511] Processing CreateTablet for tablet b977eff59197438ea161d25742031723 (DEFAULT_TABLE table=default.Table_62d1e55cf6304e208c648b8d70b41550 [id=004c240f66f44847a97ce78efa3d3cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.347471 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b977eff59197438ea161d25742031723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.347774 1610 tablet_service.cc:1511] Processing CreateTablet for tablet da698cb245ee42e6bd02e7b342035372 (DEFAULT_TABLE table=default.Table_aab8dd1a3e894677b333dc115aa2222c [id=5c6280db5c9d446f87ddf965c517583f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.348300 1601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ccedb53d6f84c0a96766fae070f191e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.348845 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da698cb245ee42e6bd02e7b342035372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.363646 1602 tablet_service.cc:1511] Processing CreateTablet for tablet 862b53262b134023ac7f990f3d3fe734 (DEFAULT_TABLE table=default.Table_4e3867de9f9848d99ecb5f0dd886d117 [id=f37873170cf1460b918bd07438190302]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.364742 1602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862b53262b134023ac7f990f3d3fe734. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:49.376297 1717 leader_election.cc:343] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:49.376642 1753 tablet_service.cc:1511] Processing CreateTablet for tablet e7db988380fe4860881532072b10eefc (DEFAULT_TABLE table=default.Table_33fb06bbd21c4209919e7b4711c86e48 [id=49870caf0d0d49059424859f3a594352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.377657 1753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7db988380fe4860881532072b10eefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.381620 1761 tablet_service.cc:1511] Processing CreateTablet for tablet d78397feb9b64336adbfe9b830e0761a (DEFAULT_TABLE table=default.Table_39b2e9330e6f46b98c77bb74fbc11ac5 [id=112fc2fc10b94fe3962001bda2d5a1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.382622 1761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d78397feb9b64336adbfe9b830e0761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.382815 1456 tablet_service.cc:1511] Processing CreateTablet for tablet 2ad83fbc8f544e7b9813774368dbb062 (DEFAULT_TABLE table=default.Table_fdbac3c4239747e396088091988a1794 [id=5585064fd41841beb6a9ccc2e30ea5af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.383795 1456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad83fbc8f544e7b9813774368dbb062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.384181 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7d6b3cf495e74057a1c5cf020b3b6d2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:49.385639 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d6b3cf495e74057a1c5cf020b3b6d2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:49.393105 1464 tablet_service.cc:1511] Processing CreateTablet for tablet de3a040b219e45ecbe2327908ff522bf (DEFAULT_TABLE table=default.Table_44c278d4bf6c4c85b644cc7b115ce1e7 [id=1447966c0f704dfcaf20082ea158f4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.394044 1464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de3a040b219e45ecbe2327908ff522bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.394330 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet bbb10f2c3d864931abb3ae2a88372d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:49.395394 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 38c3eb388d474d718554775d65f58d78 (table default.Table_392605e427d14287b4fccdd9b198c6a2 [id=d6ca68cf0d954555b422b90e6502d6cc]) successfully deleted
I20260501 13:57:49.400867 1130 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 3d0fa1a12a554984a893ec75e8b32382
I20260501 13:57:49.402555 1130 catalog_manager.cc:6741] ReplaceTablet: tablet 3d0fa1a12a554984a893ec75e8b32382 deleted and replaced by tablet efef3f3f25e24159aeaf877fafdfd105
I20260501 13:57:49.402881 1130 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41700 (request call id 434) took 2066 ms. Trace:
I20260501 13:57:49.403141 1130 rpcz_store.cc:276] 0501 13:57:47.336569 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.336697 (+ 128us) service_pool.cc:224] Handling call
0501 13:57:49.402827 (+2066130us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:48.870735 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:48.939413 (+ 68678us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:48.939438 (+ 25us) write_op.cc:435] Acquired schema lock
0501 13:57:48.939448 (+ 10us) tablet.cc:662] Decoding operations
0501 13:57:48.939750 (+ 302us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:48.939785 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:48.939806 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:57:48.940010 (+ 204us) tablet.cc:701] Row locks acquired
0501 13:57:48.940021 (+ 11us) write_op.cc:260] PREPARE: finished
0501 13:57:48.940136 (+ 115us) write_op.cc:270] Start()
0501 13:57:48.940212 (+ 76us) write_op.cc:276] Timestamp: P: 1777643868940143 usec, L: 0
0501 13:57:48.940235 (+ 23us) op_driver.cc:348] REPLICATION: starting
0501 13:57:48.940716 (+ 481us) log.cc:844] Serialized 625 byte log entry
0501 13:57:49.387405 (+446689us) op_driver.cc:464] REPLICATION: finished
0501 13:57:49.390853 (+ 3448us) write_op.cc:301] APPLY: starting
0501 13:57:49.390907 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:49.391010 (+ 103us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:49.391087 (+ 77us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:49.391992 (+ 905us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:49.392181 (+ 189us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:57:49.392222 (+ 41us) write_op.cc:312] APPLY: finished
0501 13:57:49.392391 (+ 169us) log.cc:844] Serialized 29 byte log entry
0501 13:57:49.392541 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:49.392604 (+ 63us) write_op.cc:454] Released schema lock
0501 13:57:49.392637 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:49.392866 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:49.392888 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3299,"apply.run_cpu_time_us":1980,"apply.run_wall_time_us":1980,"num_ops":2,"prepare.queue_time_us":26932,"prepare.run_cpu_time_us":1884,"prepare.run_wall_time_us":70182,"replication_time_us":447087}]]}
I20260501 13:57:49.406044 1760 tablet_service.cc:1511] Processing CreateTablet for tablet e527fe78682b4a319c8890ba52800099 (DEFAULT_TABLE table=default.Table_96a8aa986c4a42bcbc6e2655bd886c5d [id=8cd7315717a74b7d9bf8f54ef46768f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.407070 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e527fe78682b4a319c8890ba52800099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.408118 1457 tablet_service.cc:1511] Processing CreateTablet for tablet e527fe78682b4a319c8890ba52800099 (DEFAULT_TABLE table=default.Table_96a8aa986c4a42bcbc6e2655bd886c5d [id=8cd7315717a74b7d9bf8f54ef46768f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.409097 1457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e527fe78682b4a319c8890ba52800099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.409965 1455 tablet_service.cc:1511] Processing CreateTablet for tablet 4d31988fb80f42d98e5274b1d98a94e8 (DEFAULT_TABLE table=default.Table_f7500aa3d01447d08cec470916e7407d [id=65c7f93b855b44fd89d88d7dffe54266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.410868 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d31988fb80f42d98e5274b1d98a94e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.411095 1458 tablet_service.cc:1511] Processing CreateTablet for tablet e7db988380fe4860881532072b10eefc (DEFAULT_TABLE table=default.Table_33fb06bbd21c4209919e7b4711c86e48 [id=49870caf0d0d49059424859f3a594352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.412017 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7db988380fe4860881532072b10eefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.424824 1461 tablet_service.cc:1511] Processing CreateTablet for tablet 4251bbae62cf4646b2b390fec137e924 (DEFAULT_TABLE table=default.Table_7985a0b2ef3a44e6ab6fc2a8d062198f [id=7dc30fa70fa1408c9e98d0e7f6724c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.425908 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4251bbae62cf4646b2b390fec137e924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.425884 1757 tablet_service.cc:1511] Processing CreateTablet for tablet 4251bbae62cf4646b2b390fec137e924 (DEFAULT_TABLE table=default.Table_7985a0b2ef3a44e6ab6fc2a8d062198f [id=7dc30fa70fa1408c9e98d0e7f6724c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.427003 1129 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 96875c3b0f7f4421ab5d9f65ba4260a2
I20260501 13:57:49.428386 1129 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41700 (request call id 436) took 2016 ms. Trace:
I20260501 13:57:49.431957 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4251bbae62cf4646b2b390fec137e924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.428705 1129 rpcz_store.cc:276] 0501 13:57:47.411628 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.411729 (+ 101us) service_pool.cc:224] Handling call
0501 13:57:47.412863 (+ 1134us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:57:47.422945 (+ 10082us) catalog_manager.cc:2815] Modifying in-memory table state
0501 13:57:47.423090 (+ 145us) catalog_manager.cc:2824] Locking tablets
0501 13:57:48.833679 (+1410589us) catalog_manager.cc:2838] Removing table and tablets from system table
0501 13:57:49.426622 (+592943us) catalog_manager.cc:2856] Removing table from by-name map
0501 13:57:49.426816 (+ 194us) catalog_manager.cc:2872] Committing in-memory state
0501 13:57:49.426911 (+ 95us) catalog_manager.cc:2876] Aborting table tasks
0501 13:57:49.428319 (+ 1408us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:49.079528 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:49.079898 (+ 370us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:49.079922 (+ 24us) write_op.cc:435] Acquired schema lock
0501 13:57:49.079935 (+ 13us) tablet.cc:662] Decoding operations
0501 13:57:49.080227 (+ 292us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:49.080263 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:49.080285 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:57:49.080512 (+ 227us) tablet.cc:701] Row locks acquired
0501 13:57:49.080524 (+ 12us) write_op.cc:260] PREPARE: finished
0501 13:57:49.080648 (+ 124us) write_op.cc:270] Start()
0501 13:57:49.080737 (+ 89us) write_op.cc:276] Timestamp: P: 1777643869080622 usec, L: 0
0501 13:57:49.080752 (+ 15us) op_driver.cc:348] REPLICATION: starting
0501 13:57:49.081250 (+ 498us) log.cc:844] Serialized 759 byte log entry
0501 13:57:49.421068 (+339818us) op_driver.cc:464] REPLICATION: finished
0501 13:57:49.421952 (+ 884us) write_op.cc:301] APPLY: starting
0501 13:57:49.422006 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:49.422102 (+ 96us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:49.422181 (+ 79us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:49.422423 (+ 242us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:49.422578 (+ 155us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0501 13:57:49.422616 (+ 38us) write_op.cc:312] APPLY: finished
0501 13:57:49.422796 (+ 180us) log.cc:844] Serialized 29 byte log entry
0501 13:57:49.426132 (+ 3336us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:49.426200 (+ 68us) write_op.cc:454] Released schema lock
0501 13:57:49.426231 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":41856,"child_traces":[["op",{"apply.queue_time_us":721,"mutex_wait_us":40131,"num_ops":2,"prepare.queue_time_us":244125,"prepare.run_cpu_time_us":1977,"prepare.run_wall_time_us":3029,"replication_time_us":340221}]]}
I20260501 13:57:49.430325 1465 tablet_service.cc:1511] Processing CreateTablet for tablet d78397feb9b64336adbfe9b830e0761a (DEFAULT_TABLE table=default.Table_39b2e9330e6f46b98c77bb74fbc11ac5 [id=112fc2fc10b94fe3962001bda2d5a1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.433143 1465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d78397feb9b64336adbfe9b830e0761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.433933 1463 tablet_service.cc:1558] Processing DeleteTablet for tablet 96875c3b0f7f4421ab5d9f65ba4260a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:57818
W20260501 13:57:49.435031 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7165c92b5ebb415ab404584d23eb5276 has no consensus state
I20260501 13:57:49.440097 1126 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 0468f84c6d1b4440b82faebd8a33fc9f (table default.Table_c8b1ea252e544b5f81a72380b00d0b0a [id=8d69802ac4b648c185a3cc65acfa581b]). Expected version 1 got 0
I20260501 13:57:49.441220 1126 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 50ec3d3f3eb449759a8df52279b2e5e8 (table default.Table_cf75782d380e49e3b4cf46f49c816974 [id=32fd27a2bdec45698663dac24dd99b12]). Expected version 2 got 0
W20260501 13:57:49.442302 1123 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.130:35289 (request call id 79) took 1017 ms (client timeout 1000 ms). Trace:
I20260501 13:57:49.441303 1755 tablet_service.cc:1558] Processing DeleteTablet for tablet bbb10f2c3d864931abb3ae2a88372d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:49.443214 1123 rpcz_store.cc:269] 0501 13:57:48.424754 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:48.491201 (+ 66447us) service_pool.cc:224] Handling call
0501 13:57:48.550417 (+ 59216us) spinlock_profiling.cc:243] Waited 56.9 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663025470 00007f56630c431a 00007f5664dd846c 00007f5664ed23ee 00007f5662ad94a4 00007f5662ad9432 00007f5662ad9361 00007f5662ad92dc 00007f5662ad8592 00007f5660daebcc 00007f5660dae006
0501 13:57:49.442240 (+891823us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:49.087534 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:49.087890 (+ 356us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:49.087913 (+ 23us) write_op.cc:435] Acquired schema lock
0501 13:57:49.087925 (+ 12us) tablet.cc:662] Decoding operations
0501 13:57:49.088216 (+ 291us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:49.088251 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:49.088271 (+ 20us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:57:49.088494 (+ 223us) tablet.cc:701] Row locks acquired
0501 13:57:49.088507 (+ 13us) write_op.cc:260] PREPARE: finished
0501 13:57:49.088621 (+ 114us) write_op.cc:270] Start()
0501 13:57:49.088696 (+ 75us) write_op.cc:276] Timestamp: P: 1777643869088599 usec, L: 0
0501 13:57:49.088712 (+ 16us) op_driver.cc:348] REPLICATION: starting
0501 13:57:49.089158 (+ 446us) log.cc:844] Serialized 907 byte log entry
0501 13:57:49.421439 (+332281us) op_driver.cc:464] REPLICATION: finished
0501 13:57:49.434637 (+ 13198us) write_op.cc:301] APPLY: starting
0501 13:57:49.434695 (+ 58us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:49.434798 (+ 103us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:49.434881 (+ 83us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:49.435234 (+ 353us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:49.435454 (+ 220us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0501 13:57:49.435493 (+ 39us) write_op.cc:312] APPLY: finished
0501 13:57:49.435674 (+ 181us) log.cc:844] Serialized 29 byte log entry
0501 13:57:49.435829 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:49.435894 (+ 65us) write_op.cc:454] Released schema lock
0501 13:57:49.435924 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":3454,"reported_tablets":8,"spinlock_wait_cycles":130809728,"child_traces":[["op",{"apply.queue_time_us":13073,"apply.run_cpu_time_us":1283,"apply.run_wall_time_us":1458,"num_ops":2,"prepare.queue_time_us":148498,"prepare.run_cpu_time_us":1958,"prepare.run_wall_time_us":296491,"raft.queue_time_us":82,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":222,"replication_time_us":332665,"threads_started":1}]]}
I20260501 13:57:49.446419 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 4251bbae62cf4646b2b390fec137e924 (DEFAULT_TABLE table=default.Table_7985a0b2ef3a44e6ab6fc2a8d062198f [id=7dc30fa70fa1408c9e98d0e7f6724c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.447448 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4251bbae62cf4646b2b390fec137e924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.453305 1609 tablet_service.cc:1511] Processing CreateTablet for tablet d78397feb9b64336adbfe9b830e0761a (DEFAULT_TABLE table=default.Table_39b2e9330e6f46b98c77bb74fbc11ac5 [id=112fc2fc10b94fe3962001bda2d5a1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.457894 1609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d78397feb9b64336adbfe9b830e0761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.457840 1762 tablet_service.cc:1511] Processing CreateTablet for tablet b977eff59197438ea161d25742031723 (DEFAULT_TABLE table=default.Table_62d1e55cf6304e208c648b8d70b41550 [id=004c240f66f44847a97ce78efa3d3cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.458803 1762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b977eff59197438ea161d25742031723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.461553 1752 tablet_service.cc:1511] Processing CreateTablet for tablet 4d31988fb80f42d98e5274b1d98a94e8 (DEFAULT_TABLE table=default.Table_f7500aa3d01447d08cec470916e7407d [id=65c7f93b855b44fd89d88d7dffe54266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.462647 1758 tablet_service.cc:1558] Processing DeleteTablet for tablet 3d0fa1a12a554984a893ec75e8b32382 with delete_type TABLET_DATA_DELETED (replaced by tablet efef3f3f25e24159aeaf877fafdfd105) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:49.463011 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d31988fb80f42d98e5274b1d98a94e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.467096 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet bbb10f2c3d864931abb3ae2a88372d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:49.467343 2751 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f04a112d3be549d98b4ee77dbefacb7b)
I20260501 13:57:49.467944 2751 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:49.470193 1755 tablet_service.cc:1511] Processing CreateTablet for tablet de3a040b219e45ecbe2327908ff522bf (DEFAULT_TABLE table=default.Table_44c278d4bf6c4c85b644cc7b115ce1e7 [id=1447966c0f704dfcaf20082ea158f4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.477921 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de3a040b219e45ecbe2327908ff522bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.480235 1124 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_907a34a78d554ca8a54102bcf65f7ed7" } modify_external_catalogs: true
I20260501 13:57:49.480762 1124 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_907a34a78d554ca8a54102bcf65f7ed7" } modify_external_catalogs: true
I20260501 13:57:49.483132 1758 tablet_service.cc:1558] Processing DeleteTablet for tablet 96875c3b0f7f4421ab5d9f65ba4260a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:49.497639 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 396 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
W20260501 13:57:49.500494 1119 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 8b1577b8649c42b898c58eef90dee1c3 (table default.Table_586f40c4fc464137a1cc94cac7cac38e [id=b0bedf9ac0584acea2717b17cab26ace], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8b1577b8649c42b898c58eef90dee1c3 consensus state has no leader
I20260501 13:57:49.504742 1217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 396 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" is_pre_election: true
I20260501 13:57:49.506114 2451 leader_election.cc:304] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50c0932256e74399a537670bb424a1b7; no voters: d18ed9dc598e4ccaaf8633eaa45a2f51, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:49.510629 2751 leader_election.cc:290] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897)
I20260501 13:57:49.511214 2751 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260501 13:57:49.512816 1754 thread.cc:641] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.026s sys 0.041s
W20260501 13:57:49.513092 1754 thread.cc:608] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.666s user 0.026s sys 0.041s
I20260501 13:57:49.525427 2742 tablet_replica.cc:333] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:49.526276 2742 raft_consensus.cc:2243] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:49.526770 2742 raft_consensus.cc:2272] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:49.531754 2047 ts_tablet_manager.cc:1929] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:49.532140 2047 log.cc:1199] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/94abe840b1994c0cb1f7ac7f4ac09cd9
I20260501 13:57:49.532753 2047 ts_tablet_manager.cc:1950] T 94abe840b1994c0cb1f7ac7f4ac09cd9 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:49.533519 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 333) took 1037 ms. Trace:
I20260501 13:57:49.533818 2047 rpcz_store.cc:276] 0501 13:57:48.495838 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:48.497216 (+ 1378us) service_pool.cc:224] Handling call
0501 13:57:49.192609 (+695393us) ts_tablet_manager.cc:1151] Deleting tablet 94abe840b1994c0cb1f7ac7f4ac09cd9
0501 13:57:49.192645 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:49.500699 (+308054us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:49.531611 (+ 30912us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:49.533454 (+ 1843us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69178,"tablet-delete.queue_time_us":694866}
I20260501 13:57:49.534961 2047 tablet_replica.cc:333] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:49.552564 1883 tablet_bootstrap.cc:654] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:49.585366 2734 leader_election.cc:290] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:49.587913 1604 tablet_service.cc:1511] Processing CreateTablet for tablet de3a040b219e45ecbe2327908ff522bf (DEFAULT_TABLE table=default.Table_44c278d4bf6c4c85b644cc7b115ce1e7 [id=1447966c0f704dfcaf20082ea158f4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.590886 2742 ts_tablet_manager.cc:1916] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:49.610968 2747 raft_consensus.cc:493] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:49.611467 2747 raft_consensus.cc:515] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.613950 2747 leader_election.cc:290] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:49.620999 1883 tablet_bootstrap.cc:492] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:49.621409 1883 ts_tablet_manager.cc:1403] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.513s user 0.005s sys 0.005s
I20260501 13:57:49.623826 1883 raft_consensus.cc:359] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.624444 1883 raft_consensus.cc:385] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:49.624704 1883 raft_consensus.cc:740] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:49.625241 1883 consensus_queue.cc:260] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.626600 1883 ts_tablet_manager.cc:1434] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260501 13:57:49.626240 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_7bfc5bf664d54c8fa5f8735e67ba624d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:49.614331 2747 raft_consensus.cc:493] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:49.626997 2747 raft_consensus.cc:515] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.627482 1883 tablet_bootstrap.cc:492] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:49.627750 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bfc5bf664d54c8fa5f8735e67ba624d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:49.628760 2747 leader_election.cc:290] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:49.634812 1123 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_8a2507a3f60d406fbb5cc29416579789" } modify_external_catalogs: true
I20260501 13:57:49.635300 1123 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_8a2507a3f60d406fbb5cc29416579789" } modify_external_catalogs: true
I20260501 13:57:49.638226 1131 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_4b58b0177375487d90d0ce607bd08b04" } new_table_name: "default.Table_2c857decba66478795e2d54647bdae4c" modify_external_catalogs: true
I20260501 13:57:49.645872 1883 tablet_bootstrap.cc:654] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:49.659801 1128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_8343c6304e644c6f856b8f6c901ac362"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:49.661309 1128 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8343c6304e644c6f856b8f6c901ac362 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:49.672096 1883 tablet_bootstrap.cc:492] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:49.672603 1883 ts_tablet_manager.cc:1403] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.004s
I20260501 13:57:49.675060 1883 raft_consensus.cc:359] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.675844 1883 raft_consensus.cc:385] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:49.676129 1883 raft_consensus.cc:740] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:49.676888 1883 consensus_queue.cc:260] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.702136 1608 tablet_service.cc:1511] Processing CreateTablet for tablet 4d31988fb80f42d98e5274b1d98a94e8 (DEFAULT_TABLE table=default.Table_f7500aa3d01447d08cec470916e7407d [id=65c7f93b855b44fd89d88d7dffe54266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:49.707337 1883 ts_tablet_manager.cc:1434] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20260501 13:57:49.708277 1883 tablet_bootstrap.cc:492] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:49.715240 2685 raft_consensus.cc:493] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0ed1f9a545474e76a661c32bf11a7fb2)
I20260501 13:57:49.715770 2685 raft_consensus.cc:515] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.717545 2685 leader_election.cc:290] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:49.728055 1603 tablet_service.cc:1558] Processing DeleteTablet for tablet 3d0fa1a12a554984a893ec75e8b32382 with delete_type TABLET_DATA_DELETED (replaced by tablet efef3f3f25e24159aeaf877fafdfd105) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:49.728665 1603 tablet_service.cc:1558] Processing DeleteTablet for tablet 96875c3b0f7f4421ab5d9f65ba4260a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:47 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:49.728626 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 6c7e65dad084440787754e24cdd579d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c7e65dad084440787754e24cdd579d9 already in progress: creating tablet
I20260501 13:57:49.729133 1603 tablet_service.cc:1558] Processing DeleteTablet for tablet 569a0240b1274d9bb830addbaf1ec343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:38 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:49.729274 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet a000b129a6ab429889b543c20750a493 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a000b129a6ab429889b543c20750a493 already in progress: creating tablet
W20260501 13:57:49.731388 1600 consensus_meta.cc:287] T d9649a8cb0e641fcbbee1bd080a23e0b P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent flushing consensus metadata: real 0.744s user 0.009s sys 0.006s
I20260501 13:57:49.739713 1882 tablet_bootstrap.cc:492] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:49.740136 1882 ts_tablet_manager.cc:1403] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.986s user 0.013s sys 0.012s
I20260501 13:57:49.743213 2582 ts_tablet_manager.cc:1929] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:49.743780 2582 log.cc:1199] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/0c736959df334aee857dbaae0f7a1e1a
I20260501 13:57:49.744760 2582 ts_tablet_manager.cc:1950] T 0c736959df334aee857dbaae0f7a1e1a P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:49.758450 1883 tablet_bootstrap.cc:654] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:49.759627 2743 raft_consensus.cc:493] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:49.760094 2743 raft_consensus.cc:515] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.761947 2743 leader_election.cc:290] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:49.775944 2742 ts_tablet_manager.cc:1929] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:49.776391 2742 log.cc:1199] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/1ab056da506f454984a07026341c534b
W20260501 13:57:49.776855 1117 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.189:45897, user_credentials={real_user=slave}} blocked reactor thread for 73774.2us
I20260501 13:57:49.778728 2742 ts_tablet_manager.cc:1950] T 1ab056da506f454984a07026341c534b P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:49.781445 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 1ab056da506f454984a07026341c534b (table default.Table_a27ca155bb494a99bc23aefa4e6823f2 [id=ec264b057ec84337b6e8e237bb31b98d]) successfully deleted
I20260501 13:57:49.782683 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1606ce24d9654233992fd06222f3ea3e" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:49.782735 1626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "967658b1be204c638fc34983293ef5ab" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:49.782888 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2b292910bf0464cb88efe70b5b0b730" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:49.783905 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81daa397384d4afb816071ee665a1dcd" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:49.785909 1883 tablet_bootstrap.cc:492] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:49.786352 1883 ts_tablet_manager.cc:1403] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.078s user 0.022s sys 0.005s
I20260501 13:57:49.788875 1883 raft_consensus.cc:359] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.789460 1883 raft_consensus.cc:385] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:49.789755 1883 raft_consensus.cc:740] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:49.790503 1883 consensus_queue.cc:260] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.796918 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de3a040b219e45ecbe2327908ff522bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:49.806070 1883 ts_tablet_manager.cc:1434] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20260501 13:57:49.806890 1459 tablet_service.cc:1558] Processing DeleteTablet for tablet 3d0fa1a12a554984a893ec75e8b32382 with delete_type TABLET_DATA_DELETED (replaced by tablet efef3f3f25e24159aeaf877fafdfd105) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:49.808032 1883 tablet_bootstrap.cc:492] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:49.813424 1883 tablet_bootstrap.cc:654] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:49.781765 1564 consensus_peers.cc:597] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3eb6d30bed2a410db8b62fdff54a5fac. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:49.818248 2755 raft_consensus.cc:493] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:49.819188 2755 raft_consensus.cc:515] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.821034 2755 leader_election.cc:290] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
W20260501 13:57:49.827327 1719 leader_election.cc:343] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:49.827730 1719 leader_election.cc:304] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:49.828624 2755 raft_consensus.cc:2749] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:49.829645 1719 leader_election.cc:343] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:49.830341 1719 leader_election.cc:343] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:49.831074 1719 leader_election.cc:343] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:49.837597 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 94abe840b1994c0cb1f7ac7f4ac09cd9 (table default.Table_404297cf87434f66977da0de3efdc121 [id=4668751f7bc649019547c38d0f8254c4]) successfully deleted
W20260501 13:57:49.840080 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 9d38d75fcd494fc4a3420bcf15f2c70f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d38d75fcd494fc4a3420bcf15f2c70f already in progress: creating tablet
W20260501 13:57:49.841018 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet bbb10f2c3d864931abb3ae2a88372d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbb10f2c3d864931abb3ae2a88372d61 already in progress: creating tablet
W20260501 13:57:49.842133 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 3d0fa1a12a554984a893ec75e8b32382 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d0fa1a12a554984a893ec75e8b32382 already in progress: creating tablet
W20260501 13:57:49.843118 1116 catalog_manager.cc:4982] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 96875c3b0f7f4421ab5d9f65ba4260a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96875c3b0f7f4421ab5d9f65ba4260a2 already in progress: creating tablet
I20260501 13:57:49.844681 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4265d9ab3734a8b97d745dc02c6b845" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:49.846021 1627 raft_consensus.cc:2468] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 0.
I20260501 13:57:49.848683 1883 tablet_bootstrap.cc:492] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:49.848708 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_9c4a2b4c05754e6ba79c9e675c56675f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:49.849174 1883 ts_tablet_manager.cc:1403] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.002s
W20260501 13:57:49.850108 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c4a2b4c05754e6ba79c9e675c56675f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:49.851382 1883 raft_consensus.cc:359] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.851971 1883 raft_consensus.cc:385] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:49.853150 1883 raft_consensus.cc:740] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:49.853829 1883 consensus_queue.cc:260] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.856022 1883 ts_tablet_manager.cc:1434] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260501 13:57:49.856849 1883 tablet_bootstrap.cc:492] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:49.878959 1131 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce74c312b5094b60835cae9ad8e3fccd has no consensus state
I20260501 13:57:49.882081 1883 tablet_bootstrap.cc:654] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:49.878067 1882 raft_consensus.cc:359] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.883036 1882 raft_consensus.cc:385] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:49.883322 1882 raft_consensus.cc:740] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:49.890877 1882 consensus_queue.cc:260] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.417891 2720 raft_consensus.cc:493] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:49.912868 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_66ac496b624b4a509a79d0ba9243d4e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:49.914320 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66ac496b624b4a509a79d0ba9243d4e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:57:49.918572 1421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 455052us
I20260501 13:57:49.934697 2734 raft_consensus.cc:493] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:49.935150 2734 raft_consensus.cc:515] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.937369 2734 leader_election.cc:290] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:49.937769 2734 raft_consensus.cc:493] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:49.938108 2734 raft_consensus.cc:515] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.941620 2734 leader_election.cc:290] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:49.942659 1881 tablet_bootstrap.cc:492] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:49.943069 1881 ts_tablet_manager.cc:1403] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 1.108s user 0.033s sys 0.091s
I20260501 13:57:49.945261 1881 raft_consensus.cc:359] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.946008 1881 raft_consensus.cc:385] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:49.946266 1881 raft_consensus.cc:740] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:49.946915 1881 consensus_queue.cc:260] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:49.948523 1451 consensus_meta.cc:287] T 0fdf7c660b624c86ad873f8a124f5f21 P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.700s user 0.000s sys 0.006s
W20260501 13:57:49.952980 1453 consensus_meta.cc:287] T a3801a20a11543bdb2cb9a5961308c71 P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.553s user 0.004s sys 0.001s
W20260501 13:57:49.964252 1448 consensus_meta.cc:287] T 862b53262b134023ac7f990f3d3fe734 P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.604s user 0.005s sys 0.000s
I20260501 13:57:49.966384 1719 leader_election.cc:304] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:49.967411 2755 raft_consensus.cc:2804] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:49.967731 2755 raft_consensus.cc:493] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:49.968019 2755 raft_consensus.cc:3060] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:49.964653 1882 ts_tablet_manager.cc:1434] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.224s user 0.004s sys 0.004s
I20260501 13:57:49.975744 2755 raft_consensus.cc:515] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.895293 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 327) took 3222 ms. Trace:
I20260501 13:57:49.977013 2582 rpcz_store.cc:276] 0501 13:57:46.672948 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.673071 (+ 123us) service_pool.cc:224] Handling call
0501 13:57:49.230323 (+2557252us) ts_tablet_manager.cc:1151] Deleting tablet 0c736959df334aee857dbaae0f7a1e1a
0501 13:57:49.230375 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:49.347341 (+116966us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:49.740975 (+393634us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:49.895228 (+154253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20744,"tablet-delete.queue_time_us":2556703}
I20260501 13:57:49.896050 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d31988fb80f42d98e5274b1d98a94e8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:49.982019 1450 consensus_meta.cc:287] T 141738e2b87743b88492d66b2cfff2d8 P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.625s user 0.004s sys 0.000s
I20260501 13:57:49.978404 2582 tablet_replica.cc:333] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:49.982442 2755 leader_election.cc:290] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:49.983314 2582 raft_consensus.cc:2243] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:49.931471 2720 raft_consensus.cc:515] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:49.983491 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1606ce24d9654233992fd06222f3ea3e" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:49.983933 2582 raft_consensus.cc:2272] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:49.985004 2720 leader_election.cc:290] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:49.985407 2047 raft_consensus.cc:2243] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:49.985671 2582 ts_tablet_manager.cc:1916] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:49.986263 2047 raft_consensus.cc:2272] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:57:49.986323 1717 leader_election.cc:343] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:49.986771 1717 leader_election.cc:304] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:49.987656 2755 raft_consensus.cc:2749] T 1606ce24d9654233992fd06222f3ea3e P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:49.987901 2047 ts_tablet_manager.cc:1916] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:49.988900 1484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4265d9ab3734a8b97d745dc02c6b845" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
W20260501 13:57:49.990868 1717 leader_election.cc:343] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:49.991531 1480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2b292910bf0464cb88efe70b5b0b730" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:49.992871 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81daa397384d4afb816071ee665a1dcd" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:49.994319 1481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5d1afb88d14ddeb09a1c4d28324a35" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:49.996338 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4265d9ab3734a8b97d745dc02c6b845" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:49.997138 1465 tablet_service.cc:1558] Processing DeleteTablet for tablet e1d7a80ad4c4455a9ddee7a955e32635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:49.997799 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c630c25ec5c46539936ec545ff4d292" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:50.006523 1882 tablet_bootstrap.cc:492] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:50.006923 1881 ts_tablet_manager.cc:1434] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.064s user 0.006s sys 0.000s
W20260501 13:57:50.001564 1478 consensus_meta.cc:287] T e2ac522f2e4c4ab48a9b7d211c82ed9e P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.959s user 0.006s sys 0.000s
W20260501 13:57:50.009753 1717 leader_election.cc:343] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.010157 1717 leader_election.cc:304] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.011874 2755 raft_consensus.cc:2749] T a2b292910bf0464cb88efe70b5b0b730 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:49.983155 1449 consensus_meta.cc:287] T da698cb245ee42e6bd02e7b342035372 P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.594s user 0.005s sys 0.002s
W20260501 13:57:50.016324 1717 leader_election.cc:343] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.016726 1717 leader_election.cc:304] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.017062 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5d1afb88d14ddeb09a1c4d28324a35" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:50.017534 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4265d9ab3734a8b97d745dc02c6b845" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf"
I20260501 13:57:50.017707 2755 raft_consensus.cc:2749] T 81daa397384d4afb816071ee665a1dcd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:50.018041 1629 raft_consensus.cc:3060] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:50.018138 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c630c25ec5c46539936ec545ff4d292" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:50.023393 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 0c736959df334aee857dbaae0f7a1e1a (table default.Table_56892e9ecd4b4b0b9133bc936b37c1c4 [id=09422f53fb424bdbaa00282b1775947d]) successfully deleted
W20260501 13:57:50.025861 1719 leader_election.cc:343] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:50.026551 1719 leader_election.cc:343] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.028611 1629 raft_consensus.cc:2468] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ed1f9a545474e76a661c32bf11a7fb2 in term 1.
W20260501 13:57:50.033591 1418 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.028s sys 0.092s
W20260501 13:57:50.033903 1418 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.741s user 0.028s sys 0.092s
W20260501 13:57:50.039683 1717 leader_election.cc:343] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.040100 1717 leader_election.cc:304] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:50.041095 1717 leader_election.cc:343] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:50.041817 1717 leader_election.cc:343] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.042138 1717 leader_election.cc:304] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.042631 2755 raft_consensus.cc:2749] T fd5d1afb88d14ddeb09a1c4d28324a35 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:50.002436 1454 consensus_meta.cc:287] T 678de81f2adb4ae6a2fce7fbfd1b08e7 P 993b9ba18d8547f8af43167e727a6a24: Time spent flushing consensus metadata: real 0.634s user 0.007s sys 0.000s
I20260501 13:57:50.049252 2755 raft_consensus.cc:2749] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:50.057022 1883 tablet_bootstrap.cc:492] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:50.057442 1883 ts_tablet_manager.cc:1403] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.201s user 0.015s sys 0.007s
I20260501 13:57:50.062587 1719 leader_election.cc:304] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.063597 2755 raft_consensus.cc:2804] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:50.064265 2755 raft_consensus.cc:697] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Running, Role: LEADER
I20260501 13:57:50.065078 2755 consensus_queue.cc:237] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:50.065951 1563 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.807s user 0.006s sys 0.014s
I20260501 13:57:50.066551 1454 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57818 (request call id 347) took 1068 ms. Trace:
I20260501 13:57:50.066879 1454 rpcz_store.cc:276] 0501 13:57:48.997824 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:48.998603 (+ 779us) service_pool.cc:224] Handling call
0501 13:57:48.999873 (+ 1270us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0501 13:57:48.999945 (+ 72us) ts_tablet_manager.cc:682] Creating new metadata...
0501 13:57:49.292058 (+292113us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.066065 (+774007us) tablet_replica.cc:206] Creating consensus instance
0501 13:57:50.066490 (+ 425us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 13:57:50.069586 1563 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.811s user 0.006s sys 0.015s
W20260501 13:57:50.071828 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 813181us
I20260501 13:57:50.075022 1881 tablet_bootstrap.cc:492] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:50.082115 1757 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48970 (request call id 351) took 1004 ms. Trace:
I20260501 13:57:50.082638 1757 rpcz_store.cc:276] 0501 13:57:49.077228 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.425308 (+348080us) service_pool.cc:224] Handling call
0501 13:57:49.431521 (+ 6213us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0501 13:57:49.431599 (+ 78us) ts_tablet_manager.cc:682] Creating new metadata...
0501 13:57:49.810284 (+378685us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.006240 (+195956us) tablet_replica.cc:206] Creating consensus instance
0501 13:57:50.082044 (+ 75804us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260501 13:57:50.086532 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_0db06643de9249d0a23a1d5a7ae78790"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:50.088084 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0db06643de9249d0a23a1d5a7ae78790 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:50.085685 1449 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57818 (request call id 353) took 1002 ms. Trace:
I20260501 13:57:50.092185 1132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_c607fe218e37451fa9a5552a06de0768"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:50.093636 1132 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c607fe218e37451fa9a5552a06de0768 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:50.094692 1125 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_628b3fff3b534cdf8d44d255ed5f241f" } new_table_name: "default.Table_1e4ffc24acfc4ca396fc5651f5c1ef52" modify_external_catalogs: true
I20260501 13:57:50.096289 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56cee07054a64e199a0bfc26b63eca2a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:50.096868 1782 raft_consensus.cc:2468] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:50.101841 1483 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56cee07054a64e199a0bfc26b63eca2a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:50.106901 1883 raft_consensus.cc:359] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.107532 1883 raft_consensus.cc:385] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:50.107784 1883 raft_consensus.cc:740] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:50.110545 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_0848fc9554cc4ebe8553b995852bc908"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:50.111057 1781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "301fad71fe6e4e53a36216c41f2850bf" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:50.111614 1781 raft_consensus.cc:2468] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
W20260501 13:57:50.112157 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0848fc9554cc4ebe8553b995852bc908 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:50.133378 1563 leader_election.cc:304] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:50.137204 1752 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48970 (request call id 355) took 1059 ms. Trace:
I20260501 13:57:50.151201 2644 raft_consensus.cc:2804] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:50.151583 2644 raft_consensus.cc:493] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:50.151917 2644 raft_consensus.cc:3060] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:50.152004 1752 rpcz_store.cc:276] 0501 13:57:49.078142 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.460590 (+382448us) service_pool.cc:224] Handling call
0501 13:57:49.462575 (+ 1985us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0501 13:57:49.462701 (+ 126us) ts_tablet_manager.cc:682] Creating new metadata...
0501 13:57:49.959773 (+497072us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.057687 (+ 97914us) tablet_replica.cc:206] Creating consensus instance
0501 13:57:50.137153 (+ 79466us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260501 13:57:50.156123 1883 consensus_queue.cc:260] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.158169 1883 ts_tablet_manager.cc:1434] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.100s user 0.005s sys 0.001s
I20260501 13:57:50.158869 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_c3573ab598cc4732af3fe89825d4e51b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:50.160337 2644 raft_consensus.cc:515] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:50.160365 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3573ab598cc4732af3fe89825d4e51b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:50.162153 2644 leader_election.cc:290] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:50.159730 1563 leader_election.cc:304] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:50.171455 2758 raft_consensus.cc:2804] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:50.172395 2758 raft_consensus.cc:493] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:50.172719 2758 raft_consensus.cc:3060] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:50.045964 1882 tablet_bootstrap.cc:654] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:50.181782 1125 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c35118ffca2d471b9d464cb11cbe4d75 consensus state has no leader
I20260501 13:57:50.190773 2582 ts_tablet_manager.cc:1929] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:50.191176 2582 log.cc:1199] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/69626ed095bd4df7859563c176dafffc
I20260501 13:57:50.191834 2582 ts_tablet_manager.cc:1950] T 69626ed095bd4df7859563c176dafffc P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:50.194023 1883 tablet_bootstrap.cc:492] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:50.213008 1883 tablet_bootstrap.cc:654] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:50.217200 1882 tablet_bootstrap.cc:492] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:50.217643 1882 ts_tablet_manager.cc:1403] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.241s user 0.008s sys 0.002s
W20260501 13:57:50.287353 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 109673us
W20260501 13:57:50.195398 1418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 58953.2us
I20260501 13:57:50.176632 2047 ts_tablet_manager.cc:1929] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:50.291298 2047 log.cc:1199] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/0c736959df334aee857dbaae0f7a1e1a
I20260501 13:57:50.290407 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 328) took 3616 ms. Trace:
I20260501 13:57:50.196638 1482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "301fad71fe6e4e53a36216c41f2850bf" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:50.295922 2582 rpcz_store.cc:276] 0501 13:57:46.674350 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.674495 (+ 145us) service_pool.cc:224] Handling call
0501 13:57:49.978242 (+3303747us) ts_tablet_manager.cc:1151] Deleting tablet 69626ed095bd4df7859563c176dafffc
0501 13:57:49.978284 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:50.171235 (+192951us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.190641 (+ 19406us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.290340 (+ 99699us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90190,"tablet-delete.queue_time_us":3303261}
I20260501 13:57:50.106814 1449 rpcz_store.cc:276] 0501 13:57:49.082851 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.083002 (+ 151us) service_pool.cc:224] Handling call
0501 13:57:49.084383 (+ 1381us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0501 13:57:49.084454 (+ 71us) ts_tablet_manager.cc:682] Creating new metadata...
0501 13:57:49.388621 (+304167us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.085159 (+696538us) tablet_replica.cc:206] Creating consensus instance
0501 13:57:50.085589 (+ 430us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260501 13:57:50.212945 1447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57818 (request call id 355) took 1119 ms. Trace:
I20260501 13:57:50.297046 2582 tablet_replica.cc:333] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:50.297798 2582 raft_consensus.cc:2243] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:50.298233 2582 raft_consensus.cc:2272] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:50.298571 1447 rpcz_store.cc:276] 0501 13:57:49.093547 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.193755 (+100208us) service_pool.cc:224] Handling call
0501 13:57:49.265480 (+ 71725us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0501 13:57:49.265563 (+ 83us) ts_tablet_manager.cc:682] Creating new metadata...
0501 13:57:50.015860 (+750297us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.211697 (+195837us) tablet_replica.cc:206] Creating consensus instance
0501 13:57:50.212885 (+ 1188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260501 13:57:50.301657 2582 ts_tablet_manager.cc:1916] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:50.221802 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56cee07054a64e199a0bfc26b63eca2a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:50.308756 2758 raft_consensus.cc:515] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.311745 2758 leader_election.cc:290] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:50.297643 2047 ts_tablet_manager.cc:1950] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
W20260501 13:57:50.315097 1563 leader_election.cc:343] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: fc6b0cfd20174d7680f8981bb850c50f
I20260501 13:57:50.316419 2743 consensus_queue.cc:1048] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:50.320412 1882 raft_consensus.cc:359] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.321802 1882 raft_consensus.cc:385] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:50.322192 1882 raft_consensus.cc:740] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:50.323036 1882 consensus_queue.cc:260] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.338562 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_86557c4c83aa4a56b92a6963d66ca04a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:57:50.340207 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86557c4c83aa4a56b92a6963d66ca04a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:50.344277 1883 tablet_bootstrap.cc:492] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:50.344693 1883 ts_tablet_manager.cc:1403] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.151s user 0.008s sys 0.000s
I20260501 13:57:50.349787 1131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_51db366aa28a41398e299c943cf4f9f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:50.357009 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_75d087387b964c72956c5a5ff030763e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:50.350092 1883 raft_consensus.cc:359] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.364653 1883 raft_consensus.cc:385] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:50.364931 1883 raft_consensus.cc:740] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:50.366544 1883 consensus_queue.cc:260] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.327981 2778 tablet.cc:1724] T 7f9c92846d4e4255a1c8ce3835a0a18d P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 13:57:50.331493 1564 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 90360.1us
W20260501 13:57:50.363411 1563 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 44658.3us
I20260501 13:57:50.388547 1132 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "default.Table_e2a480ab55e041c5a1adc51e98337db5" } new_table_name: "default.Table_13bc55336f6e4f9d8ec006df8bb8f0aa" modify_external_catalogs: true
I20260501 13:57:50.389092 1882 ts_tablet_manager.cc:1434] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.171s user 0.003s sys 0.004s
I20260501 13:57:50.250646 1881 tablet_bootstrap.cc:654] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:50.397266 1782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56cee07054a64e199a0bfc26b63eca2a" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
W20260501 13:57:50.261885 1421 leader_election.cc:343] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Not found: Tablet not found: 0c736959df334aee857dbaae0f7a1e1a
I20260501 13:57:50.397820 1782 raft_consensus.cc:3060] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:57:50.398144 1418 leader_election.cc:343] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 0c736959df334aee857dbaae0f7a1e1a
I20260501 13:57:50.399650 1418 leader_election.cc:304] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993b9ba18d8547f8af43167e727a6a24; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf
W20260501 13:57:50.400202 1418 raft_consensus.cc:2686] T 0c736959df334aee857dbaae0f7a1e1a P 993b9ba18d8547f8af43167e727a6a24: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260501 13:57:50.400429 1418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:34859, user_credentials={real_user=slave}} blocked reactor thread for 111736us
I20260501 13:57:50.401232 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "301fad71fe6e4e53a36216c41f2850bf" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:50.401799 1783 raft_consensus.cc:3060] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:50.285980 1478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.16.130:47889 (request call id 195) took 1268 ms. Trace:
I20260501 13:57:50.404646 1883 ts_tablet_manager.cc:1434] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.057s user 0.005s sys 0.000s
I20260501 13:57:50.403457 1478 rpcz_store.cc:276] 0501 13:57:49.017333 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.041526 (+ 24193us) service_pool.cc:224] Handling call
0501 13:57:49.041980 (+ 454us) raft_consensus.cc:1471] Updating replica for 1 ops
0501 13:57:50.008046 (+966066us) raft_consensus.cc:1534] Early marking committed up to index 0
0501 13:57:50.008073 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0501 13:57:50.008500 (+ 427us) log.cc:844] Serialized 26 byte log entry
0501 13:57:50.063792 (+ 55292us) raft_consensus.cc:1645] Marking committed up to 1
0501 13:57:50.263752 (+199960us) log.cc:844] Serialized 14 byte log entry
0501 13:57:50.267234 (+ 3482us) raft_consensus.cc:1697] Filling consensus response to leader.
0501 13:57:50.285738 (+ 18504us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0501 13:57:50.285783 (+ 45us) raft_consensus.cc:1685] finished
0501 13:57:50.285798 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0501 13:57:50.285913 (+ 115us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":199962,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":24,"raft.queue_time_us":3845,"raft.run_cpu_time_us":388,"raft.run_wall_time_us":15409,"thread_start_us":223538,"threads_started":2,"wal-append.queue_time_us":55813}
W20260501 13:57:50.403445 1421 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:36735, user_credentials={real_user=slave}} blocked reactor thread for 141611us
W20260501 13:57:50.414175 1132 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef191926320a4e0fa8bd94e47e9b9a86 consensus state has no leader
I20260501 13:57:50.416115 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 334) took 1636 ms. Trace:
I20260501 13:57:50.416545 2047 rpcz_store.cc:276] 0501 13:57:48.779771 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:48.779940 (+ 169us) service_pool.cc:224] Handling call
0501 13:57:49.534815 (+754875us) ts_tablet_manager.cc:1151] Deleting tablet 0c736959df334aee857dbaae0f7a1e1a
0501 13:57:49.534855 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:49.986069 (+451214us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b54000f1254. stack: 00007f9e73a05c83 00007f9e73a05b80 00007f9e73576574 000000000050e094 000000000050e029 00007f9e7aa54951 00007f9e7aaf3742 00007f9e7af167bc 00007f9e7bb6ecca 00007f9e7bb9f990 00007f9e7bb85efd 00007f9e7bb85ea9 00007f9e7bb85e39 00007f9e7bb85e01 00007f9e7bb8502d 00000000005b3bc4
0501 13:57:50.086655 (+100586us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.176484 (+ 89829us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.416053 (+239569us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1034504448,"tablet-delete.queue_time_us":754352}
I20260501 13:57:50.418185 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet 0c736959df334aee857dbaae0f7a1e1a (table default.Table_56892e9ecd4b4b0b9133bc936b37c1c4 [id=09422f53fb424bdbaa00282b1775947d]) successfully deleted
I20260501 13:57:50.419302 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 345) took 1421 ms. Trace:
I20260501 13:57:50.420948 2755 raft_consensus.cc:493] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:50.421398 2755 raft_consensus.cc:515] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.423249 2755 leader_election.cc:290] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
W20260501 13:57:50.424476 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75d087387b964c72956c5a5ff030763e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:50.424932 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cd353ca044465a6692656412e7d82" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:50.426183 1717 leader_election.cc:343] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.431810 2743 raft_consensus.cc:493] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:50.432130 1783 raft_consensus.cc:2468] T 301fad71fe6e4e53a36216c41f2850bf P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:50.432307 2743 raft_consensus.cc:515] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.434065 2743 leader_election.cc:290] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
W20260501 13:57:50.436769 1131 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51db366aa28a41398e299c943cf4f9f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:50.444751 1782 raft_consensus.cc:2468] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:50.449867 1883 tablet_bootstrap.cc:492] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
W20260501 13:57:50.451911 1564 leader_election.cc:343] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Not found: Tablet not found: fc6b0cfd20174d7680f8981bb850c50f
I20260501 13:57:50.452292 1564 leader_election.cc:304] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.458880 2644 raft_consensus.cc:2749] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:57:50.460263 1563 leader_election.cc:304] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:50.461277 2743 raft_consensus.cc:2804] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:50.461681 2743 raft_consensus.cc:697] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:50.462404 2743 consensus_queue.cc:237] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:50.467016 1563 leader_election.cc:343] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: a2cf9528d5c24fc68e3dfdd93bf5c7bb
I20260501 13:57:50.467594 1563 leader_election.cc:304] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:50.468667 2644 raft_consensus.cc:2804] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:50.469067 2644 raft_consensus.cc:697] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:50.469661 2582 ts_tablet_manager.cc:1929] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:50.469798 2644 consensus_queue.cc:237] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:50.472818 1564 leader_election.cc:343] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.254376 1452 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57818 (request call id 350) took 1201 ms. Trace:
I20260501 13:57:50.479066 1452 rpcz_store.cc:276] 0501 13:57:49.053120 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.053303 (+ 183us) service_pool.cc:224] Handling call
0501 13:57:49.056254 (+ 2951us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0501 13:57:49.056335 (+ 81us) ts_tablet_manager.cc:682] Creating new metadata...
0501 13:57:50.013680 (+957345us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.241572 (+227892us) tablet_replica.cc:206] Creating consensus instance
0501 13:57:50.254292 (+ 12720us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260501 13:57:50.481139 1125 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "default.Table_67dfb9c64e154290bd6a095e8cdab9bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:57:50.470018 2582 log.cc:1199] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/38c3eb388d474d718554775d65f58d78
I20260501 13:57:50.419765 2047 rpcz_store.cc:276] 0501 13:57:48.997389 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.010539 (+ 13150us) service_pool.cc:224] Handling call
0501 13:57:50.418970 (+1408431us) ts_tablet_manager.cc:1151] Deleting tablet 9d38d75fcd494fc4a3420bcf15f2c70f
0501 13:57:50.419010 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:50.419248 (+ 238us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1407880}
I20260501 13:57:50.483013 2582 ts_tablet_manager.cc:1950] T 38c3eb388d474d718554775d65f58d78 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
W20260501 13:57:50.484138 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 9d38d75fcd494fc4a3420bcf15f2c70f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d38d75fcd494fc4a3420bcf15f2c70f already in progress: creating tablet
I20260501 13:57:50.484925 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 329) took 3785 ms. Trace:
W20260501 13:57:50.485538 1564 consensus_peers.cc:597] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:57:50.485900 1444 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57810 (request call id 40) took 1360 ms. Trace:
I20260501 13:57:50.487166 2755 raft_consensus.cc:493] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:50.487660 2755 raft_consensus.cc:515] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.487986 1444 rpcz_store.cc:276] 0501 13:57:49.125769 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.125943 (+ 174us) service_pool.cc:224] Handling call
0501 13:57:50.267765 (+1141822us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b5400180954. stack: 00007f9e73a05c83 00007f9e73a05b80 00007f9e73576574 000000000050e094 000000000050e029 00007f9e7aa54951 00007f9e7aae52d8 00007f9e7bb27d9d 00007f9e7a2ac104 00007f9e7a2ac092 00007f9e7a2abfc1 00007f9e7a2abf3c 00007f9e7a2ab1f2 00007f9e787debcc 00007f9e787de006 00007f9e787dd9af
0501 13:57:50.318433 (+ 50668us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b5400123328. stack: 00007f9e73a05c83 00007f9e73a05b80 00007f9e73576574 000000000050e094 000000000050e029 00007f9e7ae8d671 00007f9e7af197ac 00007f9e7bb27d70 00007f9e7a2ac104 00007f9e7a2ac092 00007f9e7a2abfc1 00007f9e7a2abf3c 00007f9e7a2ab1f2 00007f9e787debcc 00007f9e787de006 00007f9e787dd9af
0501 13:57:50.485713 (+167280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2139574400}
I20260501 13:57:50.489638 2755 leader_election.cc:290] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:50.490725 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef622e34d46a4b19a59e916d22c2b18b" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:50.492174 1717 leader_election.cc:343] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:50.499142 1564 leader_election.cc:343] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:50.499923 1564 leader_election.cc:343] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.485248 2582 rpcz_store.cc:276] 0501 13:57:46.699620 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.740035 (+ 40415us) service_pool.cc:224] Handling call
0501 13:57:50.296894 (+3556859us) ts_tablet_manager.cc:1151] Deleting tablet 38c3eb388d474d718554775d65f58d78
0501 13:57:50.296935 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:50.330206 (+ 33271us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.469532 (+139326us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.484869 (+ 15337us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3555883}
I20260501 13:57:50.502552 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 332) took 3746 ms. Trace:
I20260501 13:57:50.503298 2582 rpcz_store.cc:276] 0501 13:57:46.756595 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.756690 (+ 95us) service_pool.cc:224] Handling call
0501 13:57:50.501524 (+3744834us) ts_tablet_manager.cc:1151] Deleting tablet bd633c026195429ca69d8041809bee45
0501 13:57:50.501578 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:50.502494 (+ 916us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3744297}
I20260501 13:57:50.505064 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 333) took 3709 ms. Trace:
I20260501 13:57:50.506318 2582 rpcz_store.cc:276] 0501 13:57:46.795374 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.795875 (+ 501us) service_pool.cc:224] Handling call
0501 13:57:50.504768 (+3708893us) ts_tablet_manager.cc:1151] Deleting tablet 37f57009c05f479481b59e49062608d5
0501 13:57:50.504806 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:50.505015 (+ 209us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3708360}
I20260501 13:57:50.506636 2782 tablet.cc:1724] T 7f9c92846d4e4255a1c8ce3835a0a18d P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:50.507877 2582 tablet_replica.cc:333] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:50.509899 1883 tablet_bootstrap.cc:654] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:50.509927 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "301fad71fe6e4e53a36216c41f2850bf" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24"
I20260501 13:57:50.508600 2582 raft_consensus.cc:2243] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:50.514032 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 363) took 1121 ms. Trace:
I20260501 13:57:50.514384 2047 rpcz_store.cc:276] 0501 13:57:49.392161 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.394214 (+ 2053us) service_pool.cc:224] Handling call
0501 13:57:50.483860 (+1089646us) ts_tablet_manager.cc:1151] Deleting tablet bbb10f2c3d864931abb3ae2a88372d61
0501 13:57:50.506167 (+ 22307us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:50.513976 (+ 7809us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1089108}
I20260501 13:57:50.515805 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 367) took 1082 ms. Trace:
W20260501 13:57:50.516849 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet bbb10f2c3d864931abb3ae2a88372d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbb10f2c3d864931abb3ae2a88372d61 already in progress: creating tablet
I20260501 13:57:50.516091 2047 rpcz_store.cc:276] 0501 13:57:49.433709 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.433813 (+ 104us) service_pool.cc:224] Handling call
0501 13:57:50.515486 (+1081673us) ts_tablet_manager.cc:1151] Deleting tablet 96875c3b0f7f4421ab5d9f65ba4260a2
0501 13:57:50.515528 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:50.515754 (+ 226us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1081149}
W20260501 13:57:50.518033 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 96875c3b0f7f4421ab5d9f65ba4260a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96875c3b0f7f4421ab5d9f65ba4260a2 already in progress: creating tablet
I20260501 13:57:50.517755 2778 tablet.cc:1724] T 7d399de014d74cab8df14f44787ca2f7 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:50.525691 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2cf9528d5c24fc68e3dfdd93bf5c7bb" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:50.530898 1117 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet ef191926320a4e0fa8bd94e47e9b9a86 (table default.Table_13bc55336f6e4f9d8ec006df8bb8f0aa [id=e8b19350d702499a82f41a6cce14dbd5], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet ef191926320a4e0fa8bd94e47e9b9a86 consensus state has no leader
I20260501 13:57:50.531442 1131 master_service.cc:954] ReplaceTablet: received request to replace tablet 15a1f80cf7d24974b00c6cca7ae5783e from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:50.509159 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 69626ed095bd4df7859563c176dafffc (table default.Table_0c2d991875174eefbc2fa69108cc7082 [id=63f1fe234b1143869f21b76118b58234]) successfully deleted
W20260501 13:57:50.538643 1125 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67dfb9c64e154290bd6a095e8cdab9bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:57:50.545760 2755 raft_consensus.cc:493] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:50.546234 2755 raft_consensus.cc:515] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.548069 2755 leader_election.cc:290] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:50.549667 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b125d1a190f947a5b2ef942f04408c51" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:50.551040 1717 leader_election.cc:343] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.553879 2790 tablet.cc:1724] T 7f9c92846d4e4255a1c8ce3835a0a18d P 993b9ba18d8547f8af43167e727a6a24: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:50.554598 2582 raft_consensus.cc:2272] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:50.556571 2582 ts_tablet_manager.cc:1916] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:50.557789 1883 tablet_bootstrap.cc:492] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:50.558252 1883 ts_tablet_manager.cc:1403] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.109s user 0.010s sys 0.000s
I20260501 13:57:50.559160 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 38c3eb388d474d718554775d65f58d78 (table default.Table_392605e427d14287b4fccdd9b198c6a2 [id=d6ca68cf0d954555b422b90e6502d6cc]) successfully deleted
W20260501 13:57:50.562084 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet bd633c026195429ca69d8041809bee45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd633c026195429ca69d8041809bee45 already in progress: creating tablet
I20260501 13:57:50.561522 1090 master-stress-test.cc:577] Killing test thread 1/10
I20260501 13:57:50.567668 1090 master-stress-test.cc:579] Killed test thread 1/10
I20260501 13:57:50.567891 1090 master-stress-test.cc:577] Killing test thread 2/10
I20260501 13:57:50.560638 1883 raft_consensus.cc:359] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.573864 1882 tablet_bootstrap.cc:492] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:50.555044 2047 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57818 (request call id 366) took 1140 ms. Trace:
I20260501 13:57:50.574807 1629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cd353ca044465a6692656412e7d82" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:50.574659 2047 rpcz_store.cc:276] 0501 13:57:49.415083 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.415181 (+ 98us) service_pool.cc:224] Handling call
0501 13:57:50.554713 (+1139532us) ts_tablet_manager.cc:1151] Deleting tablet 3d0fa1a12a554984a893ec75e8b32382
0501 13:57:50.554757 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:50.554990 (+ 233us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":737647}
I20260501 13:57:50.575107 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef622e34d46a4b19a59e916d22c2b18b" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:50.576112 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b125d1a190f947a5b2ef942f04408c51" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:50.576229 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 3d0fa1a12a554984a893ec75e8b32382 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d0fa1a12a554984a893ec75e8b32382 already in progress: creating tablet
W20260501 13:57:50.595013 1564 leader_election.cc:343] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:57:50.595885 1564 leader_election.cc:343] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.596244 1564 leader_election.cc:304] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.571329 1883 raft_consensus.cc:385] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:50.566759 2782 tablet.cc:1724] T 7d399de014d74cab8df14f44787ca2f7 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:50.601189 1883 raft_consensus.cc:740] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:50.601816 1883 consensus_queue.cc:260] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:50.574193 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 37f57009c05f479481b59e49062608d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37f57009c05f479481b59e49062608d5 already in progress: creating tablet
I20260501 13:57:50.604046 1883 ts_tablet_manager.cc:1434] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.045s user 0.004s sys 0.000s
I20260501 13:57:50.605420 2047 tablet_replica.cc:333] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24: stopping tablet replica
I20260501 13:57:50.610227 1090 master-stress-test.cc:579] Killed test thread 2/10
I20260501 13:57:50.610527 1090 master-stress-test.cc:577] Killing test thread 3/10
I20260501 13:57:50.610754 1090 master-stress-test.cc:579] Killed test thread 3/10
I20260501 13:57:50.610940 1090 master-stress-test.cc:577] Killing test thread 4/10
I20260501 13:57:50.611135 1090 master-stress-test.cc:579] Killed test thread 4/10
I20260501 13:57:50.611308 1090 master-stress-test.cc:577] Killing test thread 5/10
I20260501 13:57:50.623054 2743 raft_consensus.cc:493] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:50.623524 2743 raft_consensus.cc:515] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.625197 2743 leader_election.cc:290] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:50.625945 2784 raft_consensus.cc:2749] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:50.626607 2047 raft_consensus.cc:2243] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:50.627068 2047 raft_consensus.cc:2272] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:50.627673 1883 tablet_bootstrap.cc:492] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:50.628700 2047 ts_tablet_manager.cc:1916] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:50.630407 1719 leader_election.cc:343] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.630932 1719 leader_election.cc:304] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.631595 2755 raft_consensus.cc:2749] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:50.632508 1719 leader_election.cc:343] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.632988 1719 leader_election.cc:304] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.633672 2755 raft_consensus.cc:2749] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:50.633796 1883 tablet_bootstrap.cc:654] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:50.634577 1719 leader_election.cc:343] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.634989 1719 leader_election.cc:304] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.635618 2755 raft_consensus.cc:2749] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:50.637265 1563 leader_election.cc:343] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: eaf868a5d2c24a45b183299d71f082a9
I20260501 13:57:50.640561 1881 tablet_bootstrap.cc:492] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:50.641539 1882 tablet_bootstrap.cc:654] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:50.644474 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaf868a5d2c24a45b183299d71f082a9" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:50.641002 1881 ts_tablet_manager.cc:1403] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.567s user 0.009s sys 0.022s
I20260501 13:57:50.647980 1881 raft_consensus.cc:359] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.648594 1881 raft_consensus.cc:385] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:50.648845 1881 raft_consensus.cc:740] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:50.654611 2791 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f04a112d3be549d98b4ee77dbefacb7b)
I20260501 13:57:50.655371 2791 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:50.659869 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 396 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
I20260501 13:57:50.661353 1883 tablet_bootstrap.cc:492] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:50.661916 1883 ts_tablet_manager.cc:1403] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.009s
W20260501 13:57:50.663379 1564 leader_election.cc:343] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.663756 1564 leader_election.cc:304] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.664793 1883 raft_consensus.cc:359] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.665364 1883 raft_consensus.cc:385] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:50.665236 1216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 396 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" is_pre_election: true
I20260501 13:57:50.665621 1883 raft_consensus.cc:740] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:50.666177 1883 consensus_queue.cc:260] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.658779 1881 consensus_queue.cc:260] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.668032 2784 raft_consensus.cc:2749] T eaf868a5d2c24a45b183299d71f082a9 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:50.668557 2451 leader_election.cc:304] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50c0932256e74399a537670bb424a1b7; no voters: d18ed9dc598e4ccaaf8633eaa45a2f51, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:50.691309 2791 leader_election.cc:290] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897)
I20260501 13:57:50.693259 2791 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260501 13:57:50.723495 1881 ts_tablet_manager.cc:1434] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.078s user 0.005s sys 0.000s
I20260501 13:57:50.724514 1881 tablet_bootstrap.cc:492] T 9169851f1358460d99a48cb6067ab3e3 P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
I20260501 13:57:50.737761 1883 ts_tablet_manager.cc:1434] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.076s user 0.005s sys 0.000s
I20260501 13:57:50.741820 1881 tablet_bootstrap.cc:654] T 9169851f1358460d99a48cb6067ab3e3 P 993b9ba18d8547f8af43167e727a6a24: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:50.757901 2744 raft_consensus.cc:493] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:50.758378 2744 raft_consensus.cc:515] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.760710 2755 raft_consensus.cc:493] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:50.761162 2755 raft_consensus.cc:515] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.762949 2755 leader_election.cc:290] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:50.763660 1883 tablet_bootstrap.cc:492] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:50.765789 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b3ac1f61eaf461384ae6a956ab56e62" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:50.769343 1123 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 05567fb59e424e8ea4f23d25161de158
I20260501 13:57:50.770766 1123 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41700 (request call id 475) took 1136 ms. Trace:
I20260501 13:57:50.771454 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet 05567fb59e424e8ea4f23d25161de158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:49 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:50.772593 1717 leader_election.cc:343] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.771124 1123 rpcz_store.cc:276] 0501 13:57:49.634342 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.634532 (+ 190us) service_pool.cc:224] Handling call
0501 13:57:49.635652 (+ 1120us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:57:49.638855 (+ 3203us) catalog_manager.cc:2815] Modifying in-memory table state
0501 13:57:49.638965 (+ 110us) catalog_manager.cc:2824] Locking tablets
0501 13:57:50.755295 (+1116330us) catalog_manager.cc:2838] Removing table and tablets from system table
0501 13:57:50.769007 (+ 13712us) catalog_manager.cc:2856] Removing table from by-name map
0501 13:57:50.769183 (+ 176us) catalog_manager.cc:2872] Committing in-memory state
0501 13:57:50.769263 (+ 80us) catalog_manager.cc:2876] Aborting table tasks
0501 13:57:50.770708 (+ 1445us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:50.757470 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:50.757897 (+ 427us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:50.757924 (+ 27us) write_op.cc:435] Acquired schema lock
0501 13:57:50.757937 (+ 13us) tablet.cc:662] Decoding operations
0501 13:57:50.758231 (+ 294us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:50.758270 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:50.758290 (+ 20us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:57:50.758558 (+ 268us) tablet.cc:701] Row locks acquired
0501 13:57:50.758572 (+ 14us) write_op.cc:260] PREPARE: finished
0501 13:57:50.758694 (+ 122us) write_op.cc:270] Start()
0501 13:57:50.758775 (+ 81us) write_op.cc:276] Timestamp: P: 1777643870758675 usec, L: 0
0501 13:57:50.758790 (+ 15us) op_driver.cc:348] REPLICATION: starting
0501 13:57:50.759264 (+ 474us) log.cc:844] Serialized 728 byte log entry
0501 13:57:50.766671 (+ 7407us) op_driver.cc:464] REPLICATION: finished
0501 13:57:50.767805 (+ 1134us) write_op.cc:301] APPLY: starting
0501 13:57:50.767856 (+ 51us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:50.767953 (+ 97us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:50.768030 (+ 77us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:50.768261 (+ 231us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:50.768388 (+ 127us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0501 13:57:50.768429 (+ 41us) write_op.cc:312] APPLY: finished
0501 13:57:50.768603 (+ 174us) log.cc:844] Serialized 29 byte log entry
0501 13:57:50.768741 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:50.768805 (+ 64us) write_op.cc:454] Released schema lock
0501 13:57:50.768834 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":985,"num_ops":2,"prepare.queue_time_us":418,"raft.queue_time_us":79,"raft.run_cpu_time_us":572,"raft.run_wall_time_us":571,"replication_time_us":7801}]]}
I20260501 13:57:50.776324 1452 tablet_service.cc:1558] Processing DeleteTablet for tablet 05567fb59e424e8ea4f23d25161de158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:49 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:50.780362 2582 ts_tablet_manager.cc:1929] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:50.780740 2582 log.cc:1199] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/e8efee17e2c34caf98c0ede01d2a943a
I20260501 13:57:50.781371 2582 ts_tablet_manager.cc:1950] T e8efee17e2c34caf98c0ede01d2a943a P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:50.784809 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b3ac1f61eaf461384ae6a956ab56e62" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:50.786202 1882 tablet_bootstrap.cc:492] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
W20260501 13:57:50.786432 1719 leader_election.cc:343] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.786661 1882 ts_tablet_manager.cc:1403] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.213s user 0.015s sys 0.000s
I20260501 13:57:50.786841 1719 leader_election.cc:304] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.787999 2755 raft_consensus.cc:2749] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:50.789937 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 05567fb59e424e8ea4f23d25161de158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:49 UTC) from {username='slave'} at 127.0.0.1:60694
W20260501 13:57:50.791016 1563 leader_election.cc:343] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 19c1e0c30dce488baed9e046d90f0ba4
I20260501 13:57:50.791471 2744 leader_election.cc:290] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:50.791545 1478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c1e0c30dce488baed9e046d90f0ba4" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
W20260501 13:57:50.793383 1564 leader_election.cc:343] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.794342 1564 leader_election.cc:304] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f90b7ff170b4eadb27abc9ef866ffcf; no voters: 0ed1f9a545474e76a661c32bf11a7fb2, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:50.806716 1883 tablet_bootstrap.cc:654] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:50.810487 2744 raft_consensus.cc:2749] T 19c1e0c30dce488baed9e046d90f0ba4 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:50.812417 1882 raft_consensus.cc:359] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.813017 1882 raft_consensus.cc:385] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:50.813275 1882 raft_consensus.cc:740] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:50.815260 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 339) took 3834 ms. Trace:
I20260501 13:57:50.815570 2582 rpcz_store.cc:276] 0501 13:57:46.980378 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.980493 (+ 115us) service_pool.cc:224] Handling call
0501 13:57:50.507743 (+3527250us) ts_tablet_manager.cc:1151] Deleting tablet e8efee17e2c34caf98c0ede01d2a943a
0501 13:57:50.507777 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:50.654252 (+146475us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.780212 (+125960us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:50.815194 (+ 34982us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3526705}
W20260501 13:57:50.816712 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 05567fb59e424e8ea4f23d25161de158 because the tablet was not found. No further retry: Not found: Tablet not found: 05567fb59e424e8ea4f23d25161de158
I20260501 13:57:50.817338 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet e8efee17e2c34caf98c0ede01d2a943a (table default.Table_ce141ed4961d49a8a07433b1dded9ce5 [id=cf2193b6ac264ef290b4bfb40970f215]) successfully deleted
I20260501 13:57:50.818161 2755 raft_consensus.cc:493] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:50.818642 2755 raft_consensus.cc:515] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.820461 2755 leader_election.cc:290] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:50.801815 2047 ts_tablet_manager.cc:1929] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:50.822489 2047 log.cc:1199] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/e1d7a80ad4c4455a9ddee7a955e32635
I20260501 13:57:50.823134 2047 ts_tablet_manager.cc:1950] T e1d7a80ad4c4455a9ddee7a955e32635 P 993b9ba18d8547f8af43167e727a6a24: Deleting consensus metadata
I20260501 13:57:50.816690 1882 consensus_queue.cc:260] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.826252 1129 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 81daa397384d4afb816071ee665a1dcd (table default.Table_51001cce4a264aacb220489c7c4bffd0 [id=7ab4b36ba8b8496ea524426efa573d33]). Expected version 3 got 0
I20260501 13:57:50.827071 1129 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 0468f84c6d1b4440b82faebd8a33fc9f (table default.Table_c8b1ea252e544b5f81a72380b00d0b0a [id=8d69802ac4b648c185a3cc65acfa581b]). Expected version 1 got 0
I20260501 13:57:50.827364 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0946834fac5f48a6a9976c21a45e0044" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:50.827724 1129 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 50ec3d3f3eb449759a8df52279b2e5e8 (table default.Table_cf75782d380e49e3b4cf46f49c816974 [id=32fd27a2bdec45698663dac24dd99b12]). Expected version 2 got 0
W20260501 13:57:50.828730 1719 leader_election.cc:343] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:50.836529 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 340) took 3845 ms. Trace:
I20260501 13:57:50.836864 2582 rpcz_store.cc:276] 0501 13:57:46.991020 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:46.991125 (+ 105us) service_pool.cc:224] Handling call
0501 13:57:50.836176 (+3845051us) ts_tablet_manager.cc:1151] Deleting tablet 069f9e9b9b6c40d48c9a622b73e83dec
0501 13:57:50.836218 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:50.836471 (+ 253us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3844529}
W20260501 13:57:50.837893 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 069f9e9b9b6c40d48c9a622b73e83dec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 069f9e9b9b6c40d48c9a622b73e83dec already in progress: creating tablet
I20260501 13:57:50.838627 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 56cee07054a64e199a0bfc26b63eca2a (table default.Table_9b824b59406d478884838d29a5d690f8 [id=389132d3db474d008cc0ee9d758b4fec]). Expected version 1 got 0
I20260501 13:57:50.839179 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet a4265d9ab3734a8b97d745dc02c6b845 (table default.Table_4440a2fc30004b389edb91265721ba86 [id=bd84d43e8e074a9383c9843cd13e7a65]). Expected version 1 got 0
I20260501 13:57:50.839699 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 301fad71fe6e4e53a36216c41f2850bf (table default.Table_9fec3571b60d4a7da2653ad41120b531 [id=70e8e791a366491eaae840dbd7565b34]). Expected version 1 got 0
W20260501 13:57:50.842410 1128 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.130:35289 (request call id 80) took 946 ms (client timeout 1000 ms). Trace:
W20260501 13:57:50.843024 1128 rpcz_store.cc:269] 0501 13:57:49.895839 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.895972 (+ 133us) service_pool.cc:224] Handling call
0501 13:57:50.842350 (+946378us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":3}
I20260501 13:57:50.844636 1883 tablet_bootstrap.cc:492] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:50.845968 1883 ts_tablet_manager.cc:1403] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.082s user 0.010s sys 0.000s
I20260501 13:57:50.848325 1883 raft_consensus.cc:359] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.848924 1883 raft_consensus.cc:385] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:50.849170 1883 raft_consensus.cc:740] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:50.850737 1390 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 05567fb59e424e8ea4f23d25161de158
I20260501 13:57:50.852859 1882 ts_tablet_manager.cc:1434] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.058s user 0.005s sys 0.000s
I20260501 13:57:50.857713 1881 tablet_bootstrap.cc:492] T 9169851f1358460d99a48cb6067ab3e3 P 993b9ba18d8547f8af43167e727a6a24: No bootstrap required, opened a new log
I20260501 13:57:50.858130 1881 ts_tablet_manager.cc:1403] T 9169851f1358460d99a48cb6067ab3e3 P 993b9ba18d8547f8af43167e727a6a24: Time spent bootstrapping tablet: real 0.134s user 0.007s sys 0.001s
I20260501 13:57:50.851914 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 0468f84c6d1b4440b82faebd8a33fc9f (table default.Table_c8b1ea252e544b5f81a72380b00d0b0a [id=8d69802ac4b648c185a3cc65acfa581b]). Expected version 1 got 0
I20260501 13:57:50.858886 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet fd5d1afb88d14ddeb09a1c4d28324a35 (table default.Table_6097a4730b784d39885bff5f7673d66f [id=5c600e77298a4757a2801c398370d5d3]). Expected version 1 got 0
I20260501 13:57:50.859854 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 81daa397384d4afb816071ee665a1dcd (table default.Table_51001cce4a264aacb220489c7c4bffd0 [id=7ab4b36ba8b8496ea524426efa573d33]). Expected version 3 got 0
I20260501 13:57:50.860513 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 50ec3d3f3eb449759a8df52279b2e5e8 (table default.Table_cf75782d380e49e3b4cf46f49c816974 [id=32fd27a2bdec45698663dac24dd99b12]). Expected version 2 got 0
I20260501 13:57:50.860394 1881 raft_consensus.cc:359] T 9169851f1358460d99a48cb6067ab3e3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.860989 1881 raft_consensus.cc:385] T 9169851f1358460d99a48cb6067ab3e3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:50.861246 1881 raft_consensus.cc:740] T 9169851f1358460d99a48cb6067ab3e3 P 993b9ba18d8547f8af43167e727a6a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993b9ba18d8547f8af43167e727a6a24, State: Initialized, Role: FOLLOWER
I20260501 13:57:50.862323 2582 tablet_replica.cc:333] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:50.862850 1882 tablet_bootstrap.cc:492] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:50.863091 2582 raft_consensus.cc:2243] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:50.866581 2582 raft_consensus.cc:2272] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:50.868387 2582 ts_tablet_manager.cc:1916] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:50.868041 1883 consensus_queue.cc:260] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.870960 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet 05567fb59e424e8ea4f23d25161de158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:49 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:50.871168 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet 05567fb59e424e8ea4f23d25161de158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:49 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:50.872743 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 05567fb59e424e8ea4f23d25161de158 because the tablet was not found. No further retry: Not found: Tablet not found: 05567fb59e424e8ea4f23d25161de158
I20260501 13:57:50.870998 1125 catalog_manager.cc:5671] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: HEALTHY } } }
I20260501 13:57:50.884507 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet a4265d9ab3734a8b97d745dc02c6b845 (table default.Table_4440a2fc30004b389edb91265721ba86 [id=bd84d43e8e074a9383c9843cd13e7a65]). Expected version 1 got 0
I20260501 13:57:50.885910 1882 tablet_bootstrap.cc:654] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:50.893692 1752 tablet_service.cc:1511] Processing CreateTablet for tablet faf8205072b14aceaa95658fd3093754 (DEFAULT_TABLE table=default.Table_0edc079d8a294a99aaca7a5847ae718d [id=9957ac58495648b5b4ecadc8419fce9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:50.894761 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf8205072b14aceaa95658fd3093754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:50.894832 1604 tablet_service.cc:1511] Processing CreateTablet for tablet faf8205072b14aceaa95658fd3093754 (DEFAULT_TABLE table=default.Table_0edc079d8a294a99aaca7a5847ae718d [id=9957ac58495648b5b4ecadc8419fce9f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:50.895496 1127 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50ec3d3f3eb449759a8df52279b2e5e8 consensus state has no leader
I20260501 13:57:50.895984 1127 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41700 (request call id 471) took 1945 ms. Trace:
W20260501 13:57:50.898447 1126 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.131:42895 (request call id 79) took 2330 ms (client timeout 1000 ms). Trace:
I20260501 13:57:50.898830 1757 tablet_service.cc:1558] Processing DeleteTablet for tablet 8dbb619863dd405ab20ead6bbebf4d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:50.898880 1126 rpcz_store.cc:269] 0501 13:57:48.568478 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:48.614258 (+ 45780us) service_pool.cc:224] Handling call
0501 13:57:50.898383 (+2284125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8456,"reported_tablets":10}
I20260501 13:57:50.899431 1760 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ab056da506f454984a07026341c534b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:40 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:50.896255 1127 rpcz_store.cc:276] 0501 13:57:48.950887 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.059508 (+108621us) service_pool.cc:224] Handling call
0501 13:57:49.085313 (+ 25805us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:57:49.085994 (+ 681us) catalog_manager.cc:3725] Apply alter schema
0501 13:57:49.130628 (+ 44634us) catalog_manager.cc:3757] Acquired catalog manager lock
0501 13:57:49.130824 (+ 196us) catalog_manager.cc:3915] Updating metadata on disk
0501 13:57:49.707101 (+576277us) catalog_manager.cc:3946] Committing alterations to in-memory state
0501 13:57:50.895927 (+1188826us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:49.491710 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:49.492065 (+ 355us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:49.492088 (+ 23us) write_op.cc:435] Acquired schema lock
0501 13:57:49.492101 (+ 13us) tablet.cc:662] Decoding operations
0501 13:57:49.492329 (+ 228us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:49.492361 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:49.492382 (+ 21us) tablet.cc:685] Acquiring locks for 1 operations
0501 13:57:49.492514 (+ 132us) tablet.cc:701] Row locks acquired
0501 13:57:49.492525 (+ 11us) write_op.cc:260] PREPARE: finished
0501 13:57:49.492628 (+ 103us) write_op.cc:270] Start()
0501 13:57:49.492708 (+ 80us) write_op.cc:276] Timestamp: P: 1777643869492642 usec, L: 0
0501 13:57:49.492722 (+ 14us) op_driver.cc:348] REPLICATION: starting
0501 13:57:49.621080 (+128358us) log.cc:844] Serialized 405 byte log entry
0501 13:57:49.703699 (+ 82619us) op_driver.cc:464] REPLICATION: finished
0501 13:57:49.704150 (+ 451us) write_op.cc:301] APPLY: starting
0501 13:57:49.704207 (+ 57us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:49.704299 (+ 92us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:49.704395 (+ 96us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:49.704567 (+ 172us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:49.704708 (+ 141us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:57:49.704750 (+ 42us) write_op.cc:312] APPLY: finished
0501 13:57:49.704933 (+ 183us) log.cc:844] Serialized 23 byte log entry
0501 13:57:49.705086 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:49.705147 (+ 61us) write_op.cc:454] Released schema lock
0501 13:57:49.705177 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:49.705407 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:49.705431 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":309,"apply.run_cpu_time_us":1223,"apply.run_wall_time_us":1222,"num_ops":1,"prepare.queue_time_us":359804,"prepare.run_cpu_time_us":1840,"prepare.run_wall_time_us":129676,"raft.queue_time_us":11106,"raft.run_cpu_time_us":786,"raft.run_wall_time_us":786,"replication_time_us":210871}]]}
I20260501 13:57:50.902040 1090 master-stress-test.cc:579] Killed test thread 5/10
I20260501 13:57:50.902281 1090 master-stress-test.cc:577] Killing test thread 6/10
I20260501 13:57:50.902638 1090 master-stress-test.cc:579] Killed test thread 6/10
I20260501 13:57:50.902837 1090 master-stress-test.cc:577] Killing test thread 7/10
I20260501 13:57:50.903259 1090 master-stress-test.cc:579] Killed test thread 7/10
I20260501 13:57:50.903445 1090 master-stress-test.cc:577] Killing test thread 8/10
I20260501 13:57:50.904059 2795 tablet_replica.cc:333] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:50.907099 1883 ts_tablet_manager.cc:1434] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.061s user 0.001s sys 0.005s
I20260501 13:57:50.907511 2795 raft_consensus.cc:2243] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:50.908057 1883 tablet_bootstrap.cc:492] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:50.908066 2795 raft_consensus.cc:2272] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:50.910724 2795 ts_tablet_manager.cc:1916] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:50.912732 2778 tablet.cc:1724] T 33a355897309420c97b1257afaefe8d1 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:50.914882 2782 tablet.cc:1724] T 33a355897309420c97b1257afaefe8d1 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 13:57:50.915727 1129 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.16.131:42895 (request call id 80) took 1224 ms (client timeout 1000 ms). Trace:
W20260501 13:57:50.916043 1129 rpcz_store.cc:269] 0501 13:57:49.691753 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.691874 (+ 121us) service_pool.cc:224] Handling call
0501 13:57:50.915649 (+1223775us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":12}
I20260501 13:57:50.917016 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 4b3ac1f61eaf461384ae6a956ab56e62
I20260501 13:57:50.918423 1124 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41700 (request call id 473) took 1657 ms. Trace:
I20260501 13:57:50.919303 1760 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b3ac1f61eaf461384ae6a956ab56e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:49 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:50.923036 1608 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b3ac1f61eaf461384ae6a956ab56e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:49 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:50.919624 1124 rpcz_store.cc:276] 0501 13:57:49.260558 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.411362 (+150804us) service_pool.cc:224] Handling call
0501 13:57:49.480119 (+ 68757us) spinlock_profiling.cc:243] Waited 67.6 ms on lock 0x7b5400030554. stack: 00007f565bfd5c83 00007f565bfd5b80 00007f565bb46574 000000000050e094 000000000050e029 00007f5663025470 00007f56630c431a 00007f5664dd846c 00007f5664ed3c46 00007f5662ae5024 00007f5662ae4fb2 00007f5662ae4ee1 00007f5662ae4e5c 00007f5662ae4112 00007f5660daebcc 00007f5660dae006
0501 13:57:49.481113 (+ 994us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:57:49.481584 (+ 471us) catalog_manager.cc:2815] Modifying in-memory table state
0501 13:57:49.481708 (+ 124us) catalog_manager.cc:2824] Locking tablets
0501 13:57:50.795163 (+1313455us) catalog_manager.cc:2838] Removing table and tablets from system table
0501 13:57:50.812741 (+ 17578us) catalog_manager.cc:2856] Removing table from by-name map
0501 13:57:50.812925 (+ 184us) catalog_manager.cc:2872] Committing in-memory state
0501 13:57:50.916927 (+104002us) catalog_manager.cc:2876] Aborting table tasks
0501 13:57:50.918368 (+ 1441us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:57:50.797048 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:57:50.797426 (+ 378us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:57:50.797449 (+ 23us) write_op.cc:435] Acquired schema lock
0501 13:57:50.797462 (+ 13us) tablet.cc:662] Decoding operations
0501 13:57:50.797757 (+ 295us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:57:50.797798 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0501 13:57:50.797820 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:57:50.798060 (+ 240us) tablet.cc:701] Row locks acquired
0501 13:57:50.798073 (+ 13us) write_op.cc:260] PREPARE: finished
0501 13:57:50.798198 (+ 125us) write_op.cc:270] Start()
0501 13:57:50.798293 (+ 95us) write_op.cc:276] Timestamp: P: 1777643870798176 usec, L: 0
0501 13:57:50.798311 (+ 18us) op_driver.cc:348] REPLICATION: starting
0501 13:57:50.798834 (+ 523us) log.cc:844] Serialized 728 byte log entry
0501 13:57:50.808229 (+ 9395us) op_driver.cc:464] REPLICATION: finished
0501 13:57:50.811407 (+ 3178us) write_op.cc:301] APPLY: starting
0501 13:57:50.811467 (+ 60us) tablet.cc:1367] starting BulkCheckPresence
0501 13:57:50.811575 (+ 108us) tablet.cc:1370] finished BulkCheckPresence
0501 13:57:50.811663 (+ 88us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:57:50.811925 (+ 262us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:57:50.812060 (+ 135us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0501 13:57:50.812101 (+ 41us) write_op.cc:312] APPLY: finished
0501 13:57:50.812289 (+ 188us) log.cc:844] Serialized 29 byte log entry
0501 13:57:50.812434 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:50.812501 (+ 67us) write_op.cc:454] Released schema lock
0501 13:57:50.812540 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0501 13:57:50.817276 (+ 4736us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:57:50.817301 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":155558784,"child_traces":[["op",{"apply.queue_time_us":2925,"apply.run_cpu_time_us":1363,"apply.run_wall_time_us":1772,"num_ops":2,"prepare.queue_time_us":116,"prepare.run_cpu_time_us":2124,"prepare.run_wall_time_us":20217,"raft.queue_time_us":90,"raft.run_cpu_time_us":636,"raft.run_wall_time_us":637,"replication_time_us":9836}]]}
I20260501 13:57:50.925058 1090 master-stress-test.cc:579] Killed test thread 8/10
I20260501 13:57:50.925304 1090 master-stress-test.cc:577] Killing test thread 9/10
I20260501 13:57:50.925561 1090 master-stress-test.cc:579] Killed test thread 9/10
I20260501 13:57:50.925750 1090 master-stress-test.cc:577] Killing test thread 10/10
I20260501 13:57:50.934161 1883 tablet_bootstrap.cc:654] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:50.965155 1128 catalog_manager.cc:5671] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:50.966507 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 56cee07054a64e199a0bfc26b63eca2a (table default.Table_9b824b59406d478884838d29a5d690f8 [id=389132d3db474d008cc0ee9d758b4fec]). Expected version 1 got 0
I20260501 13:57:50.970209 1608 tablet_service.cc:1511] Processing CreateTablet for tablet 05567fb59e424e8ea4f23d25161de158 (DEFAULT_TABLE table=default.Table_8a2507a3f60d406fbb5cc29416579789 [id=d7cce6e962fb427ca73a5701beaeff91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:50.973345 1752 tablet_service.cc:1511] Processing CreateTablet for tablet 05567fb59e424e8ea4f23d25161de158 (DEFAULT_TABLE table=default.Table_8a2507a3f60d406fbb5cc29416579789 [id=d7cce6e962fb427ca73a5701beaeff91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:50.974438 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05567fb59e424e8ea4f23d25161de158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:50.986264 1117 catalog_manager.cc:5002] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): tablet e1d7a80ad4c4455a9ddee7a955e32635 (table default.Table_cf80c30d6b014d56aab7e294515801d5 [id=16dcfdbf585e45b29a6cdc6442d0990b]) successfully deleted
W20260501 13:57:50.986984 1117 catalog_manager.cc:4969] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 05567fb59e424e8ea4f23d25161de158 because the tablet was not found. No further retry: Not found: Tablet not found: 05567fb59e424e8ea4f23d25161de158
I20260501 13:57:50.988458 1882 tablet_bootstrap.cc:492] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:50.988885 1882 ts_tablet_manager.cc:1403] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.126s user 0.015s sys 0.003s
I20260501 13:57:50.991266 1882 raft_consensus.cc:359] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:50.991889 1882 raft_consensus.cc:385] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:50.995710 1591 tablet_service.cc:1511] Processing CreateTablet for tablet d17a8b0bac524e67b619071657156342 (DEFAULT_TABLE table=default.Table_88ada069a99749fea12f89d823a4e95a [id=bc1f8b8d5fd74afcad0a03708169b5e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.005955 1447 tablet_service.cc:1511] Processing CreateTablet for tablet faf8205072b14aceaa95658fd3093754 (DEFAULT_TABLE table=default.Table_0edc079d8a294a99aaca7a5847ae718d [id=9957ac58495648b5b4ecadc8419fce9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.006574 1128 catalog_manager.cc:5671] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:51.007591 1447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf8205072b14aceaa95658fd3093754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.007820 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 301fad71fe6e4e53a36216c41f2850bf (table default.Table_9fec3571b60d4a7da2653ad41120b531 [id=70e8e791a366491eaae840dbd7565b34]). Expected version 1 got 0
I20260501 13:57:51.008951 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 406e1af8b7d9488f88697206daa87c05 (table default.Table_759f3009542147de94c73784e1698246 [id=0f7c81f8d9a64f17921007cc8f9f3ba5]). Expected version 1 got 0
I20260501 13:57:51.015583 1593 tablet_service.cc:1511] Processing CreateTablet for tablet f5128fbba70e4b21813523ecda67a1d1 (DEFAULT_TABLE table=default.Table_e095439dec874189b8a4ea72e4dc1b48 [id=a5039d210ece4246a59b506e9c767afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.016503 2582 ts_tablet_manager.cc:1929] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:51.016870 2582 log.cc:1199] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/a2cf9528d5c24fc68e3dfdd93bf5c7bb
I20260501 13:57:51.017570 2582 ts_tablet_manager.cc:1950] T a2cf9528d5c24fc68e3dfdd93bf5c7bb P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:51.018790 1449 tablet_service.cc:1511] Processing CreateTablet for tablet 05567fb59e424e8ea4f23d25161de158 (DEFAULT_TABLE table=default.Table_8a2507a3f60d406fbb5cc29416579789 [id=d7cce6e962fb427ca73a5701beaeff91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.020009 1449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05567fb59e424e8ea4f23d25161de158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.023113 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05567fb59e424e8ea4f23d25161de158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.026592 1760 tablet_service.cc:1511] Processing CreateTablet for tablet d17a8b0bac524e67b619071657156342 (DEFAULT_TABLE table=default.Table_88ada069a99749fea12f89d823a4e95a [id=bc1f8b8d5fd74afcad0a03708169b5e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.030984 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf8205072b14aceaa95658fd3093754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.038194 1757 tablet_service.cc:1511] Processing CreateTablet for tablet f5128fbba70e4b21813523ecda67a1d1 (DEFAULT_TABLE table=default.Table_e095439dec874189b8a4ea72e4dc1b48 [id=a5039d210ece4246a59b506e9c767afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.038996 1452 tablet_service.cc:1558] Processing DeleteTablet for tablet 05567fb59e424e8ea4f23d25161de158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:49 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:51.040035 1454 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b3ac1f61eaf461384ae6a956ab56e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:49 UTC) from {username='slave'} at 127.0.0.1:57818
I20260501 13:57:50.992139 1882 raft_consensus.cc:740] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:51.042853 1457 tablet_service.cc:1511] Processing CreateTablet for tablet d17a8b0bac524e67b619071657156342 (DEFAULT_TABLE table=default.Table_88ada069a99749fea12f89d823a4e95a [id=bc1f8b8d5fd74afcad0a03708169b5e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.042713 1882 consensus_queue.cc:260] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.043936 1457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17a8b0bac524e67b619071657156342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.044199 1124 master_service.cc:954] ReplaceTablet: received request to replace tablet 6f6cd353ca044465a6692656412e7d82 from {username='slave'} at 127.0.0.1:41700
I20260501 13:57:51.044586 1882 ts_tablet_manager.cc:1434] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.055s user 0.002s sys 0.003s
I20260501 13:57:51.045634 1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17a8b0bac524e67b619071657156342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.046301 1458 tablet_service.cc:1511] Processing CreateTablet for tablet f5128fbba70e4b21813523ecda67a1d1 (DEFAULT_TABLE table=default.Table_e095439dec874189b8a4ea72e4dc1b48 [id=a5039d210ece4246a59b506e9c767afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.047667 1458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5128fbba70e4b21813523ecda67a1d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.052268 1593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5128fbba70e4b21813523ecda67a1d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.052515 1882 tablet_bootstrap.cc:492] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:51.065536 1882 tablet_bootstrap.cc:654] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:51.074721 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17a8b0bac524e67b619071657156342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.076261 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5128fbba70e4b21813523ecda67a1d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.086261 1485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0946834fac5f48a6a9976c21a45e0044" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993b9ba18d8547f8af43167e727a6a24" is_pre_election: true
I20260501 13:57:51.110212 2755 raft_consensus.cc:493] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:51.110577 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 33a355897309420c97b1257afaefe8d1 (table default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76]). Expected version 1 got 0
I20260501 13:57:51.110728 2755 raft_consensus.cc:515] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.112609 2755 leader_election.cc:290] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:51.113045 2755 raft_consensus.cc:493] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:51.113409 2755 raft_consensus.cc:515] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.114852 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bb7fdcad3a54317b43bfd58f08e89d0" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:51.116241 1719 leader_election.cc:343] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:51.116819 1131 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_f9c99e1d322d44258e1845eeb858e579 [id=1eefeb189133435f960daacad0e615cf] alter complete (version 1)
I20260501 13:57:51.117302 1125 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_aa47824eda84455d9d52df61514a253e [id=81117c514ffd4ab6a8d4dde20d7559e4] alter complete (version 1)
I20260501 13:57:51.119747 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 342) took 3937 ms. Trace:
I20260501 13:57:51.120347 1883 tablet_bootstrap.cc:492] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:51.120043 2582 rpcz_store.cc:276] 0501 13:57:47.182804 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.182936 (+ 132us) service_pool.cc:224] Handling call
0501 13:57:50.853889 (+3670953us) ts_tablet_manager.cc:1151] Deleting tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb
0501 13:57:50.862192 (+ 8303us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:50.978634 (+116442us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:51.002370 (+ 23736us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:51.119690 (+117320us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3670389}
I20260501 13:57:51.120723 1883 ts_tablet_manager.cc:1403] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.213s user 0.008s sys 0.004s
I20260501 13:57:51.121411 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet a2cf9528d5c24fc68e3dfdd93bf5c7bb (table default.Table_e7f17dee34594eb3a711ffbf706d4e59 [id=8570b73bebda4640a91c1bc894942e38]) successfully deleted
I20260501 13:57:51.123417 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 344) took 3849 ms. Trace:
I20260501 13:57:51.123391 1883 raft_consensus.cc:359] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.123710 2582 rpcz_store.cc:276] 0501 13:57:47.274111 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.274231 (+ 120us) service_pool.cc:224] Handling call
0501 13:57:51.123091 (+3848860us) ts_tablet_manager.cc:1151] Deleting tablet 81230beeff8d434aa2dac467c35d9703
0501 13:57:51.123130 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:51.123362 (+ 232us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3847935}
I20260501 13:57:51.124366 1883 raft_consensus.cc:385] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:51.124696 1883 raft_consensus.cc:740] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
W20260501 13:57:51.124727 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 81230beeff8d434aa2dac467c35d9703 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81230beeff8d434aa2dac467c35d9703 already in progress: creating tablet
I20260501 13:57:51.125363 1883 consensus_queue.cc:260] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.127326 1883 ts_tablet_manager.cc:1434] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260501 13:57:50.861755 1881 consensus_queue.cc:260] T 9169851f1358460d99a48cb6067ab3e3 P 993b9ba18d8547f8af43167e727a6a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.132011 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daca20eeafb141f2bf8b4233e0c8ea3b" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:51.132217 1881 ts_tablet_manager.cc:1434] T 9169851f1358460d99a48cb6067ab3e3 P 993b9ba18d8547f8af43167e727a6a24: Time spent starting tablet: real 0.274s user 0.005s sys 0.000s
I20260501 13:57:51.132877 2755 leader_election.cc:290] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:51.133309 1881 tablet_bootstrap.cc:492] T e8efee17e2c34caf98c0ede01d2a943a P 993b9ba18d8547f8af43167e727a6a24: Bootstrap starting.
W20260501 13:57:51.133234 1719 leader_election.cc:343] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:51.138101 1883 tablet_bootstrap.cc:492] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:51.147778 1606 tablet_service.cc:1511] Processing CreateTablet for tablet 4b2decaf756f4a96859047b49574fd3b (DEFAULT_TABLE table=default.Table_208156cfc4324941b36574ed9fb2c398 [id=53a6f983523147bc93ee87734a6aa002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.148798 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b2decaf756f4a96859047b49574fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.156298 2795 ts_tablet_manager.cc:1929] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:51.160058 1598 tablet_service.cc:1511] Processing CreateTablet for tablet c339d10bb45e43e6be1f5aab03eb7265 (DEFAULT_TABLE table=default.Table_1b0b21c8b69f49239f2fdeeac464d4ff [id=b44b81242a004f73bc946b0cb8782e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.161201 1598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c339d10bb45e43e6be1f5aab03eb7265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.166859 2795 log.cc:1199] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/8dbb619863dd405ab20ead6bbebf4d69
I20260501 13:57:51.171732 2795 ts_tablet_manager.cc:1950] T 8dbb619863dd405ab20ead6bbebf4d69 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:51.098997 1454 tablet_service.cc:1511] Processing CreateTablet for tablet 4b2decaf756f4a96859047b49574fd3b (DEFAULT_TABLE table=default.Table_208156cfc4324941b36574ed9fb2c398 [id=53a6f983523147bc93ee87734a6aa002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.161303 1752 tablet_service.cc:1511] Processing CreateTablet for tablet 4b2decaf756f4a96859047b49574fd3b (DEFAULT_TABLE table=default.Table_208156cfc4324941b36574ed9fb2c398 [id=53a6f983523147bc93ee87734a6aa002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.184873 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 8dbb619863dd405ab20ead6bbebf4d69 (table default.Table_6ee51a9fc8ae4340b14c506cf9c450c2 [id=e61fe8a70f66497db17db0afacc35100]) successfully deleted
W20260501 13:57:51.185540 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 1ab056da506f454984a07026341c534b because the tablet was not found. No further retry: Not found: Tablet not found: 1ab056da506f454984a07026341c534b
W20260501 13:57:51.186759 1117 catalog_manager.cc:4969] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 05567fb59e424e8ea4f23d25161de158 because the tablet was not found. No further retry: Not found: Tablet not found: 05567fb59e424e8ea4f23d25161de158
W20260501 13:57:51.187368 1117 catalog_manager.cc:4982] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): delete failed for tablet 4b3ac1f61eaf461384ae6a956ab56e62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b3ac1f61eaf461384ae6a956ab56e62 already in progress: creating tablet
I20260501 13:57:51.195475 2805 consensus_queue.cc:1048] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:51.199023 2805 raft_consensus.cc:493] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:51.199617 2805 raft_consensus.cc:515] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.203110 1883 tablet_bootstrap.cc:654] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:51.204854 1624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b0ed592fd3642c194bb946c8c6003b4" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:51.207257 1124 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: Sending DeleteTablet for 3 replicas of tablet 6f6cd353ca044465a6692656412e7d82
I20260501 13:57:51.209084 1124 catalog_manager.cc:6741] ReplaceTablet: tablet 6f6cd353ca044465a6692656412e7d82 deleted and replaced by tablet ee3c6b8c310f4c809afa816b871e01bc
I20260501 13:57:51.209578 1596 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f6cd353ca044465a6692656412e7d82 with delete_type TABLET_DATA_DELETED (replaced by tablet ee3c6b8c310f4c809afa816b871e01bc) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:51.213984 1090 master-stress-test.cc:579] Killed test thread 10/10
I20260501 13:57:51.214358 1090 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu with pid 1373
I20260501 13:57:51.227072 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 345) took 3952 ms. Trace:
I20260501 13:57:51.205976 2795 tablet_replica.cc:333] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
W20260501 13:57:51.207645 1719 leader_election.cc:343] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:51.236920 1757 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f6cd353ca044465a6692656412e7d82 with delete_type TABLET_DATA_DELETED (replaced by tablet ee3c6b8c310f4c809afa816b871e01bc) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:51.237404 1883 tablet_bootstrap.cc:492] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:51.237815 1883 ts_tablet_manager.cc:1403] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.000s
I20260501 13:57:51.237952 1748 tablet_service.cc:1558] Processing DeleteTablet for tablet daca20eeafb141f2bf8b4233e0c8ea3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:51.208804 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b2decaf756f4a96859047b49574fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.238547 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet b125d1a190f947a5b2ef942f04408c51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:51.238538 1759 tablet_service.cc:1558] Processing DeleteTablet for tablet ef622e34d46a4b19a59e916d22c2b18b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:51.239032 1744 tablet_service.cc:1511] Processing CreateTablet for tablet c339d10bb45e43e6be1f5aab03eb7265 (DEFAULT_TABLE table=default.Table_1b0b21c8b69f49239f2fdeeac464d4ff [id=b44b81242a004f73bc946b0cb8782e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.233860 2582 rpcz_store.cc:276] 0501 13:57:47.274824 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.274918 (+ 94us) service_pool.cc:224] Handling call
0501 13:57:51.226732 (+3951814us) ts_tablet_manager.cc:1151] Deleting tablet bdc412d7b5634ef2893b59467be4fa92
0501 13:57:51.226778 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:51.227014 (+ 236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19288,"tablet-delete.queue_time_us":3951341}
W20260501 13:57:51.240639 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet bdc412d7b5634ef2893b59467be4fa92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdc412d7b5634ef2893b59467be4fa92 already in progress: creating tablet
I20260501 13:57:51.241823 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 343) took 3970 ms. Trace:
I20260501 13:57:51.242112 2582 rpcz_store.cc:276] 0501 13:57:47.270951 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.274678 (+ 3727us) service_pool.cc:224] Handling call
0501 13:57:51.240749 (+3966071us) ts_tablet_manager.cc:1151] Deleting tablet 6df58010b9084f4ab4710dd8eb5c4041
0501 13:57:51.241527 (+ 778us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:51.241768 (+ 241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23440,"spinlock_wait_cycles":1608064,"tablet-delete.queue_time_us":3961737}
W20260501 13:57:51.243999 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 6df58010b9084f4ab4710dd8eb5c4041 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6df58010b9084f4ab4710dd8eb5c4041 already in progress: creating tablet
I20260501 13:57:51.246135 1883 raft_consensus.cc:359] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.246814 1883 raft_consensus.cc:385] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:51.247053 1883 raft_consensus.cc:740] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:51.251691 1759 tablet_service.cc:1511] Processing CreateTablet for tablet f48709ba7dd5491aab339f2fa3a74bfe (DEFAULT_TABLE table=default.Table_ccc1648285ff4873a78a3ee0632630d9 [id=f5f6324cbccc47fea90fe2692e5ac8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.252765 2582 tablet_replica.cc:333] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:51.254407 1606 tablet_service.cc:1511] Processing CreateTablet for tablet f48709ba7dd5491aab339f2fa3a74bfe (DEFAULT_TABLE table=default.Table_ccc1648285ff4873a78a3ee0632630d9 [id=f5f6324cbccc47fea90fe2692e5ac8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.255404 1606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f48709ba7dd5491aab339f2fa3a74bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.213054 2805 leader_election.cc:290] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:51.256914 1760 tablet_service.cc:1558] Processing DeleteTablet for tablet 8dbb619863dd405ab20ead6bbebf4d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:41 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:51.249246 1883 consensus_queue.cc:260] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.257643 1757 tablet_service.cc:1558] Processing DeleteTablet for tablet 1c630c25ec5c46539936ec545ff4d292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:51.259178 1883 ts_tablet_manager.cc:1434] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260501 13:57:51.258257 2582 raft_consensus.cc:2243] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:51.251752 1596 tablet_service.cc:1558] Processing DeleteTablet for tablet 19c1e0c30dce488baed9e046d90f0ba4 with delete_type TABLET_DATA_DELETED (replaced by tablet ef191926320a4e0fa8bd94e47e9b9a86) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:51.259747 2582 raft_consensus.cc:2272] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:51.265592 1757 tablet_service.cc:1511] Processing CreateTablet for tablet d00a953434c24a898fda5a5591643e33 (DEFAULT_TABLE table=default.Table_66ca9758ef054c48bc7e719d382f285e [id=3e38e1d88be0457baa0d0aab705c0ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.307876 1604 tablet_service.cc:1558] Processing DeleteTablet for tablet eaf868a5d2c24a45b183299d71f082a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:34 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:51.307937 1608 tablet_service.cc:1511] Processing CreateTablet for tablet 332632fa748e4d0794a45f26132f77c8 (DEFAULT_TABLE table=default.Table_5e24fbb6bd82479b9f7ff4911a582bd1 [id=e85f20c6dc224a3fb931bc3ea4459e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.308943 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332632fa748e4d0794a45f26132f77c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.313545 1604 tablet_service.cc:1511] Processing CreateTablet for tablet f1c8556f0170415781f70129d824594d (DEFAULT_TABLE table=default.Table_72bc40e077c7414c96936f5aa8eb7511 [id=ffb96cdfd41641e7bf20dd5e5f803cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.316723 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c8556f0170415781f70129d824594d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.266677 1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c339d10bb45e43e6be1f5aab03eb7265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.324015 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d00a953434c24a898fda5a5591643e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.270212 2795 raft_consensus.cc:2243] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:51.326830 2795 raft_consensus.cc:2272] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:51.293381 1763 tablet_service.cc:1511] Processing CreateTablet for tablet f1c8556f0170415781f70129d824594d (DEFAULT_TABLE table=default.Table_72bc40e077c7414c96936f5aa8eb7511 [id=ffb96cdfd41641e7bf20dd5e5f803cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.327930 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c8556f0170415781f70129d824594d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.327711 1596 tablet_service.cc:1511] Processing CreateTablet for tablet d00a953434c24a898fda5a5591643e33 (DEFAULT_TABLE table=default.Table_66ca9758ef054c48bc7e719d382f285e [id=3e38e1d88be0457baa0d0aab705c0ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.328768 1596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d00a953434c24a898fda5a5591643e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.293824 1883 tablet_bootstrap.cc:492] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:51.311306 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 332632fa748e4d0794a45f26132f77c8 (DEFAULT_TABLE table=default.Table_5e24fbb6bd82479b9f7ff4911a582bd1 [id=e85f20c6dc224a3fb931bc3ea4459e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.331362 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332632fa748e4d0794a45f26132f77c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.336719 2795 ts_tablet_manager.cc:1916] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:51.337797 1595 tablet_service.cc:1511] Processing CreateTablet for tablet 7b30bfb3e2fc477b9159a9dd6e1aeb8b (DEFAULT_TABLE table=default.Table_a56a7addc3cd4e4aab56ab751ddcb8c1 [id=70568e2ceb5b4ecb8805756be9b0bd9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.338575 2758 consensus_queue.cc:1048] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:51.341346 1882 tablet_bootstrap.cc:492] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:51.341483 1595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b30bfb3e2fc477b9159a9dd6e1aeb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.341771 1882 ts_tablet_manager.cc:1403] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.289s user 0.012s sys 0.004s
I20260501 13:57:51.342648 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f48709ba7dd5491aab339f2fa3a74bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.359707 1883 tablet_bootstrap.cc:654] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:51.366904 1125 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_785faee6678c46f1afa60872f1582c7f [id=84b84a980ef34f0292ef7b9488a09b76] alter complete (version 1)
I20260501 13:57:51.371397 2582 ts_tablet_manager.cc:1916] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:51.373037 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet 1bb7fdcad3a54317b43bfd58f08e89d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b6b99b30bf14f7e88af8a52c3aa7fa9) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:51.383924 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b0ed592fd3642c194bb946c8c6003b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:44 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:51.392454 2795 ts_tablet_manager.cc:1929] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:51.392962 2795 log.cc:1199] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/4b3ac1f61eaf461384ae6a956ab56e62
I20260501 13:57:51.393697 2795 ts_tablet_manager.cc:1950] T 4b3ac1f61eaf461384ae6a956ab56e62 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:51.399952 1883 tablet_bootstrap.cc:492] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:51.400370 1883 ts_tablet_manager.cc:1403] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.107s user 0.009s sys 0.000s
I20260501 13:57:51.402710 1883 raft_consensus.cc:359] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.403329 1883 raft_consensus.cc:385] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:51.403560 1883 raft_consensus.cc:740] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:51.404161 1883 consensus_queue.cc:260] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.406026 1883 ts_tablet_manager.cc:1434] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260501 13:57:51.406878 1883 tablet_bootstrap.cc:492] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:51.408906 1882 raft_consensus.cc:359] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.409538 1882 raft_consensus.cc:385] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:51.409855 1882 raft_consensus.cc:740] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:51.410490 1882 consensus_queue.cc:260] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.412364 1882 ts_tablet_manager.cc:1434] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.070s user 0.005s sys 0.000s
I20260501 13:57:51.413146 1882 tablet_bootstrap.cc:492] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:51.415890 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet a4265d9ab3734a8b97d745dc02c6b845 (table default.Table_4440a2fc30004b389edb91265721ba86 [id=bd84d43e8e074a9383c9843cd13e7a65]). Expected version 1 got 0
I20260501 13:57:51.431023 1593 tablet_service.cc:1558] Processing DeleteTablet for tablet f32170902c134c93a9aa71b2530c845e with delete_type TABLET_DATA_DELETED (replaced by tablet 3b0ed592fd3642c194bb946c8c6003b4) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:51.435145 2778 tablet.cc:1724] T a4265d9ab3734a8b97d745dc02c6b845 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:51.449410 1883 tablet_bootstrap.cc:654] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:51.452312 1744 tablet_service.cc:1511] Processing CreateTablet for tablet 7b30bfb3e2fc477b9159a9dd6e1aeb8b (DEFAULT_TABLE table=default.Table_a56a7addc3cd4e4aab56ab751ddcb8c1 [id=70568e2ceb5b4ecb8805756be9b0bd9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.453419 1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b30bfb3e2fc477b9159a9dd6e1aeb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.454408 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 56cee07054a64e199a0bfc26b63eca2a (table default.Table_9b824b59406d478884838d29a5d690f8 [id=389132d3db474d008cc0ee9d758b4fec]). Expected version 1 got 0
I20260501 13:57:51.455092 1593 tablet_service.cc:1511] Processing CreateTablet for tablet 9f677dcbf0f540749ff71906240b703b (DEFAULT_TABLE table=default.Table_64024529e14042c3be91a36b033ada90 [id=48266956ca8b4274aea505435692daaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.456177 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 4b3ac1f61eaf461384ae6a956ab56e62 (table default.Table_907a34a78d554ca8a54102bcf65f7ed7 [id=65f926c9bcbd48fe898fe450f021dae1]) successfully deleted
I20260501 13:57:51.457792 2795 tablet_replica.cc:333] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:51.458655 2795 raft_consensus.cc:2243] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:51.459118 2795 raft_consensus.cc:2272] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:51.472715 1763 tablet_service.cc:1558] Processing DeleteTablet for tablet 1042bdf58c1f488daeaec284eb3319d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:37 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:51.472653 2816 tablet.cc:1724] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:51.474133 2582 ts_tablet_manager.cc:1929] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:51.474548 2582 log.cc:1199] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/fc6b0cfd20174d7680f8981bb850c50f
I20260501 13:57:51.475217 2582 ts_tablet_manager.cc:1950] T fc6b0cfd20174d7680f8981bb850c50f P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:51.484139 2795 ts_tablet_manager.cc:1916] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:51.489106 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 9f677dcbf0f540749ff71906240b703b (DEFAULT_TABLE table=default.Table_64024529e14042c3be91a36b033ada90 [id=48266956ca8b4274aea505435692daaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.497094 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f677dcbf0f540749ff71906240b703b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.510828 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 56cee07054a64e199a0bfc26b63eca2a (table default.Table_9b824b59406d478884838d29a5d690f8 [id=389132d3db474d008cc0ee9d758b4fec]). Expected version 1 got 0
I20260501 13:57:51.519304 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 967658b1be204c638fc34983293ef5ab (table default.Table_1ec94dd4afbb40d19a6f68aaefad8790 [id=f57f3fcda9444fa0a1ad062576f8d051]). Expected version 3 got 0
I20260501 13:57:51.519945 1128 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 56cee07054a64e199a0bfc26b63eca2a (table default.Table_9b824b59406d478884838d29a5d690f8 [id=389132d3db474d008cc0ee9d758b4fec]). Expected version 1 got 0
I20260501 13:57:51.523383 1757 tablet_service.cc:1511] Processing CreateTablet for tablet 7165c92b5ebb415ab404584d23eb5276 (DEFAULT_TABLE table=default.Table_6b367195dbab461e9a822d9ee5facdc1 [id=c3ebb8bfcc984ac3bfa74b7d3a461b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.524809 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7165c92b5ebb415ab404584d23eb5276. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.525386 1598 tablet_service.cc:1511] Processing CreateTablet for tablet 7165c92b5ebb415ab404584d23eb5276 (DEFAULT_TABLE table=default.Table_6b367195dbab461e9a822d9ee5facdc1 [id=c3ebb8bfcc984ac3bfa74b7d3a461b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.529606 2755 raft_consensus.cc:493] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:51.530053 2755 raft_consensus.cc:515] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.531849 2755 leader_election.cc:290] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:51.533737 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f7db275918c41139cf9bf44bba001a7" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:51.534966 1719 leader_election.cc:343] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:51.539640 2755 raft_consensus.cc:493] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:51.539882 1883 tablet_bootstrap.cc:492] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:51.540253 1883 ts_tablet_manager.cc:1403] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.134s user 0.009s sys 0.000s
I20260501 13:57:51.540076 2755 raft_consensus.cc:515] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:51.541940 1117 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet ef191926320a4e0fa8bd94e47e9b9a86 (table default.Table_13bc55336f6e4f9d8ec006df8bb8f0aa [id=e8b19350d702499a82f41a6cce14dbd5], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet ef191926320a4e0fa8bd94e47e9b9a86 consensus state has no leader
I20260501 13:57:51.542706 1883 raft_consensus.cc:359] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.543346 2755 leader_election.cc:290] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:51.548451 1125 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_4440a2fc30004b389edb91265721ba86 [id=bd84d43e8e074a9383c9843cd13e7a65] alter complete (version 1)
I20260501 13:57:51.549901 2795 ts_tablet_manager.cc:1929] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:51.550290 2795 log.cc:1199] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/6f6cd353ca044465a6692656412e7d82
I20260501 13:57:51.550491 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a7eecc7345949cc81ad73e2d0370c9c" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:51.550989 2795 ts_tablet_manager.cc:1950] T 6f6cd353ca044465a6692656412e7d82 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
W20260501 13:57:51.551842 1719 leader_election.cc:343] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:51.543309 1883 raft_consensus.cc:385] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:51.553485 1883 raft_consensus.cc:740] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:51.554080 1883 consensus_queue.cc:260] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.567814 1883 ts_tablet_manager.cc:1434] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260501 13:57:51.579118 1759 tablet_service.cc:1511] Processing CreateTablet for tablet 352c10a8418c4ac4a3caaef3c08ef07a (DEFAULT_TABLE table=default.Table_52acb3d1a0d1450bb41440f4de75ac91 [id=7b430735e8b640dc84cf92f308f20362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.585628 2778 tablet.cc:1724] T 56cee07054a64e199a0bfc26b63eca2a P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:51.587344 2755 raft_consensus.cc:493] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:51.588028 1593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f677dcbf0f540749ff71906240b703b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.587836 2755 raft_consensus.cc:515] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.589743 2755 leader_election.cc:290] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:51.592015 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 346) took 4090 ms. Trace:
I20260501 13:57:51.592092 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31c73582788648aab55d678ad6e4fc76" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
I20260501 13:57:51.592393 2582 rpcz_store.cc:276] 0501 13:57:47.501428 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.501559 (+ 131us) service_pool.cc:224] Handling call
0501 13:57:51.252596 (+3751037us) ts_tablet_manager.cc:1151] Deleting tablet fc6b0cfd20174d7680f8981bb850c50f
0501 13:57:51.252654 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:51.423386 (+170732us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:51.473991 (+ 50605us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:51.591960 (+117969us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4730,"tablet-delete.queue_time_us":3625816}
I20260501 13:57:51.593465 2582 tablet_replica.cc:333] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
W20260501 13:57:51.593428 1719 leader_election.cc:343] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:51.594131 2582 raft_consensus.cc:2243] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:57:51.594746 2743 raft_consensus.cc:493] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:51.595312 2743 raft_consensus.cc:515] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.595813 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet fc6b0cfd20174d7680f8981bb850c50f (table default.Table_f3ef319b6f34419d987e039dd1c2eb54 [id=c7df1385355c46c0896e9586953edf34]) successfully deleted
I20260501 13:57:51.595098 1598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7165c92b5ebb415ab404584d23eb5276. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.597158 2743 leader_election.cc:290] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:51.600059 1608 tablet_service.cc:1511] Processing CreateTablet for tablet 352c10a8418c4ac4a3caaef3c08ef07a (DEFAULT_TABLE table=default.Table_52acb3d1a0d1450bb41440f4de75ac91 [id=7b430735e8b640dc84cf92f308f20362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.601085 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352c10a8418c4ac4a3caaef3c08ef07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.601486 2582 raft_consensus.cc:2272] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:51.603536 2582 ts_tablet_manager.cc:1916] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:51.610301 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6cd353ca044465a6692656412e7d82" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
W20260501 13:57:51.611788 1563 leader_election.cc:343] T 6f6cd353ca044465a6692656412e7d82 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Illegal state: Tablet not RUNNING: SHUTDOWN
I20260501 13:57:51.614979 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 6f6cd353ca044465a6692656412e7d82 (table default.Table_95c99739e2574437b16501060ce816af [id=a6953bc50bab4e7f92adfee0ef034465]) successfully deleted
I20260501 13:57:51.616501 2795 tablet_replica.cc:333] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:51.617379 2795 raft_consensus.cc:2243] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:51.617866 2795 raft_consensus.cc:2272] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:51.619777 2795 ts_tablet_manager.cc:1916] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:51.621945 2816 tablet.cc:1724] T 56cee07054a64e199a0bfc26b63eca2a P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:51.630695 1882 tablet_bootstrap.cc:654] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:51.633582 1759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352c10a8418c4ac4a3caaef3c08ef07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.633697 1883 tablet_bootstrap.cc:492] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:51.634765 2758 raft_consensus.cc:493] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:51.635226 2758 raft_consensus.cc:515] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.637125 2758 leader_election.cc:290] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:51.638686 1783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "967658b1be204c638fc34983293ef5ab" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" is_pre_election: true
I20260501 13:57:51.639262 1783 raft_consensus.cc:2468] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 0.
I20260501 13:57:51.640396 1563 leader_election.cc:304] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:51.641522 2758 raft_consensus.cc:2804] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:57:51.641839 2758 raft_consensus.cc:493] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:57:51.642130 2758 raft_consensus.cc:3060] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:51.647269 1595 tablet_service.cc:1511] Processing CreateTablet for tablet d7323bf442804229a1b5cafdf00bc166 (DEFAULT_TABLE table=default.Table_8722bc86cc204923b96c371a00b62e75 [id=f3a9a38e8c824425a3bec2993c16711f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.648496 1595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7323bf442804229a1b5cafdf00bc166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.653849 1883 tablet_bootstrap.cc:654] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:51.666265 1125 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_9b824b59406d478884838d29a5d690f8 [id=389132d3db474d008cc0ee9d758b4fec] alter complete (version 1)
I20260501 13:57:51.676192 1744 tablet_service.cc:1511] Processing CreateTablet for tablet d7323bf442804229a1b5cafdf00bc166 (DEFAULT_TABLE table=default.Table_8722bc86cc204923b96c371a00b62e75 [id=f3a9a38e8c824425a3bec2993c16711f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.677412 1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7323bf442804229a1b5cafdf00bc166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.681929 2743 consensus_queue.cc:1048] T 301fad71fe6e4e53a36216c41f2850bf P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:57:51.682600 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 5ac6d366015f4762b0a34b3ce7b2ea9b (DEFAULT_TABLE table=default.Table_70a73112cf374a6db6a56bcc5faf040d [id=1a16c4c37080484a8b186f747262cf30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.683653 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ac6d366015f4762b0a34b3ce7b2ea9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.684996 1596 tablet_service.cc:1511] Processing CreateTablet for tablet 5ac6d366015f4762b0a34b3ce7b2ea9b (DEFAULT_TABLE table=default.Table_70a73112cf374a6db6a56bcc5faf040d [id=1a16c4c37080484a8b186f747262cf30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.689553 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 82dbd8c60f65483eaf176c3f205096c6 (DEFAULT_TABLE table=default.Table_5c18f34cdebc4aba990d32e590b96317 [id=bb1b978c29034a6bae86ba6af97f152e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.710562 2755 raft_consensus.cc:493] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:51.711037 2755 raft_consensus.cc:515] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.711791 2758 raft_consensus.cc:515] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.712352 2582 ts_tablet_manager.cc:1929] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:57:51.712690 2582 log.cc:1199] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/3eb6d30bed2a410db8b62fdff54a5fac
I20260501 13:57:51.712977 2755 leader_election.cc:290] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:51.713377 2582 ts_tablet_manager.cc:1950] T 3eb6d30bed2a410db8b62fdff54a5fac P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting consensus metadata
I20260501 13:57:51.710516 1883 tablet_bootstrap.cc:492] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:51.714803 1883 ts_tablet_manager.cc:1403] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.006s
I20260501 13:57:51.717186 1883 raft_consensus.cc:359] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.717839 1883 raft_consensus.cc:385] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:51.718088 1883 raft_consensus.cc:740] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:51.722788 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1042bdf58c1f488daeaec284eb3319d5" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:51.724228 1719 leader_election.cc:343] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:51.725111 1883 consensus_queue.cc:260] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.726707 2795 ts_tablet_manager.cc:1929] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:51.727087 2795 log.cc:1199] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/daca20eeafb141f2bf8b4233e0c8ea3b
I20260501 13:57:51.727687 1883 ts_tablet_manager.cc:1434] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260501 13:57:51.727735 2795 ts_tablet_manager.cc:1950] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:51.727764 1596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ac6d366015f4762b0a34b3ce7b2ea9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.728615 1883 tablet_bootstrap.cc:492] T 82d94a00960a4abeb3ddc5f1c1b2d3cd P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:51.728785 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82dbd8c60f65483eaf176c3f205096c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.729256 2795 tablet_replica.cc:333] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:51.730078 2795 raft_consensus.cc:2243] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:51.730552 2795 raft_consensus.cc:2272] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:51.732836 1882 tablet_bootstrap.cc:492] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:51.733214 1882 ts_tablet_manager.cc:1403] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.320s user 0.011s sys 0.000s
I20260501 13:57:51.733191 2795 ts_tablet_manager.cc:1916] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:51.735126 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet daca20eeafb141f2bf8b4233e0c8ea3b (table default.Table_430d0393c555431e8d75edfd188a80d6 [id=9bb928e7ec304e699ff70bd51b9cdfaa]) successfully deleted
I20260501 13:57:51.736447 2758 leader_election.cc:290] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Requested vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:51.736366 1779 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "967658b1be204c638fc34983293ef5ab" candidate_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed1f9a545474e76a661c32bf11a7fb2"
I20260501 13:57:51.736917 1779 raft_consensus.cc:3060] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:57:51.740890 1883 tablet_bootstrap.cc:654] T 82d94a00960a4abeb3ddc5f1c1b2d3cd P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:51.743510 1779 raft_consensus.cc:2468] T 967658b1be204c638fc34983293ef5ab P 0ed1f9a545474e76a661c32bf11a7fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f90b7ff170b4eadb27abc9ef866ffcf in term 1.
I20260501 13:57:51.744773 1563 leader_election.cc:304] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2, 8f90b7ff170b4eadb27abc9ef866ffcf; no voters:
I20260501 13:57:51.745684 2758 raft_consensus.cc:2804] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:57:51.746639 2758 raft_consensus.cc:697] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [term 1 LEADER]: Becoming Leader. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Running, Role: LEADER
I20260501 13:57:51.749329 1882 raft_consensus.cc:359] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.749997 1882 raft_consensus.cc:385] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:51.768894 1744 tablet_service.cc:1511] Processing CreateTablet for tablet 82dbd8c60f65483eaf176c3f205096c6 (DEFAULT_TABLE table=default.Table_5c18f34cdebc4aba990d32e590b96317 [id=bb1b978c29034a6bae86ba6af97f152e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.770009 1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82dbd8c60f65483eaf176c3f205096c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.747408 2758 consensus_queue.cc:237] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.785965 1598 tablet_service.cc:1511] Processing CreateTablet for tablet ddb88b05b6104e6c80f9d2a0e02a4895 (DEFAULT_TABLE table=default.Table_17529a8d7af544bab22d0471bfea0f0a [id=5276f98c47894e4cac4e5b95544446f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.788827 1763 tablet_service.cc:1511] Processing CreateTablet for tablet ddb88b05b6104e6c80f9d2a0e02a4895 (DEFAULT_TABLE table=default.Table_17529a8d7af544bab22d0471bfea0f0a [id=5276f98c47894e4cac4e5b95544446f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.789849 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb88b05b6104e6c80f9d2a0e02a4895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.797840 2755 raft_consensus.cc:493] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:51.798307 2755 raft_consensus.cc:515] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.800189 2755 leader_election.cc:290] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:51.800213 1125 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 301fad71fe6e4e53a36216c41f2850bf (table default.Table_9fec3571b60d4a7da2653ad41120b531 [id=70e8e791a366491eaae840dbd7565b34]). Expected version 1 got 0
I20260501 13:57:51.750247 1882 raft_consensus.cc:740] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:51.815102 1882 consensus_queue.cc:260] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.817296 2795 ts_tablet_manager.cc:1929] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:51.817979 2795 log.cc:1199] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/b125d1a190f947a5b2ef942f04408c51
I20260501 13:57:51.818621 1882 ts_tablet_manager.cc:1434] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.085s user 0.007s sys 0.000s
I20260501 13:57:51.819002 2795 ts_tablet_manager.cc:1950] T b125d1a190f947a5b2ef942f04408c51 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:51.819634 1627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77c59fc1fdd645af8c3017b6836c8bf4" candidate_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" is_pre_election: true
W20260501 13:57:51.821358 1719 leader_election.cc:343] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:57:51.822119 1591 tablet_service.cc:1511] Processing CreateTablet for tablet 67eb4cc6eeba4862b55366c78109b145 (DEFAULT_TABLE table=default.Table_c79a50afdbec462cb118e9f092e962f0 [id=147b591d37b64969b7d4a3f23bd905e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.822422 1757 tablet_service.cc:1511] Processing CreateTablet for tablet 67eb4cc6eeba4862b55366c78109b145 (DEFAULT_TABLE table=default.Table_c79a50afdbec462cb118e9f092e962f0 [id=147b591d37b64969b7d4a3f23bd905e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.825115 1591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67eb4cc6eeba4862b55366c78109b145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.827324 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 347) took 4121 ms. Trace:
I20260501 13:57:51.827669 2582 rpcz_store.cc:276] 0501 13:57:47.706216 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:47.706324 (+ 108us) service_pool.cc:224] Handling call
0501 13:57:51.593326 (+3887002us) ts_tablet_manager.cc:1151] Deleting tablet 3eb6d30bed2a410db8b62fdff54a5fac
0501 13:57:51.593365 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:51.656634 (+ 63269us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:51.712234 (+ 55600us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:51.827264 (+115030us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3869737}
I20260501 13:57:51.828702 1119 catalog_manager.cc:5002] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): tablet 3eb6d30bed2a410db8b62fdff54a5fac (table default.Table_52acb3d1a0d1450bb41440f4de75ac91 [id=7b430735e8b640dc84cf92f308f20362]) successfully deleted
I20260501 13:57:51.830616 2743 raft_consensus.cc:493] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:51.831164 2743 raft_consensus.cc:515] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:51.835736 1563 leader_election.cc:343] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: f32170902c134c93a9aa71b2530c845e
I20260501 13:57:51.834357 2743 leader_election.cc:290] T f32170902c134c93a9aa71b2530c845e P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:51.838438 1598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb88b05b6104e6c80f9d2a0e02a4895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.843858 1882 tablet_bootstrap.cc:492] T f52ea8fb8ae0427d804001aed42cd67c P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
I20260501 13:57:51.843900 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 359) took 2894 ms. Trace:
I20260501 13:57:51.845575 2582 rpcz_store.cc:276] 0501 13:57:48.950994 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:48.951097 (+ 103us) service_pool.cc:224] Handling call
0501 13:57:51.843544 (+2892447us) ts_tablet_manager.cc:1151] Deleting tablet 9d38d75fcd494fc4a3420bcf15f2c70f
0501 13:57:51.843591 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:51.843843 (+ 252us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2891810}
I20260501 13:57:51.846730 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 378) took 2535 ms. Trace:
W20260501 13:57:51.846823 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 9d38d75fcd494fc4a3420bcf15f2c70f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d38d75fcd494fc4a3420bcf15f2c70f already in progress: creating tablet
I20260501 13:57:51.847018 2582 rpcz_store.cc:276] 0501 13:57:49.311124 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.466978 (+155854us) service_pool.cc:224] Handling call
0501 13:57:51.846390 (+2379412us) ts_tablet_manager.cc:1151] Deleting tablet bbb10f2c3d864931abb3ae2a88372d61
0501 13:57:51.846426 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:51.846682 (+ 256us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2378907}
I20260501 13:57:51.848088 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 382) took 2120 ms. Trace:
I20260501 13:57:51.848378 2582 rpcz_store.cc:276] 0501 13:57:49.727613 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.727933 (+ 320us) service_pool.cc:224] Handling call
0501 13:57:51.847775 (+2119842us) ts_tablet_manager.cc:1151] Deleting tablet 3d0fa1a12a554984a893ec75e8b32382
0501 13:57:51.847810 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:51.848042 (+ 232us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2119294}
W20260501 13:57:51.848788 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet bbb10f2c3d864931abb3ae2a88372d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbb10f2c3d864931abb3ae2a88372d61 already in progress: creating tablet
I20260501 13:57:51.849411 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 13f02c141dc24f2186267c48f5ee3db6 (DEFAULT_TABLE table=default.Table_133c504302a64fe0b6133ad5cfe65de1 [id=93619ebd5a7d477cbc0d7b681b9ecbdc]), partition=RANGE (key) PARTITION UNBOUNDED
W20260501 13:57:51.849591 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 3d0fa1a12a554984a893ec75e8b32382 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d0fa1a12a554984a893ec75e8b32382 already in progress: creating tablet
I20260501 13:57:51.850828 1604 tablet_service.cc:1511] Processing CreateTablet for tablet 13f02c141dc24f2186267c48f5ee3db6 (DEFAULT_TABLE table=default.Table_133c504302a64fe0b6133ad5cfe65de1 [id=93619ebd5a7d477cbc0d7b681b9ecbdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.852588 1593 tablet_service.cc:1511] Processing CreateTablet for tablet dc485efda1414befa957251d6a7de298 (DEFAULT_TABLE table=default.Table_d22b3187808547e88c2cfb3f9b22c917 [id=db09dbc0620444de811191b495221de6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.853160 1752 tablet_service.cc:1511] Processing CreateTablet for tablet dc485efda1414befa957251d6a7de298 (DEFAULT_TABLE table=default.Table_d22b3187808547e88c2cfb3f9b22c917 [id=db09dbc0620444de811191b495221de6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.854094 1604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f02c141dc24f2186267c48f5ee3db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.854700 1593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc485efda1414befa957251d6a7de298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.859447 1883 tablet_bootstrap.cc:492] T 82d94a00960a4abeb3ddc5f1c1b2d3cd P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:51.859869 1883 ts_tablet_manager.cc:1403] T 82d94a00960a4abeb3ddc5f1c1b2d3cd P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.131s user 0.011s sys 0.016s
I20260501 13:57:51.872189 2582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60694 (request call id 383) took 2144 ms. Trace:
I20260501 13:57:51.872557 2582 rpcz_store.cc:276] 0501 13:57:49.727918 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:49.728544 (+ 626us) service_pool.cc:224] Handling call
0501 13:57:51.871839 (+2143295us) ts_tablet_manager.cc:1151] Deleting tablet 96875c3b0f7f4421ab5d9f65ba4260a2
0501 13:57:51.871883 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:51.872136 (+ 253us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2142838}
W20260501 13:57:51.873539 1119 catalog_manager.cc:4982] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): delete failed for tablet 96875c3b0f7f4421ab5d9f65ba4260a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96875c3b0f7f4421ab5d9f65ba4260a2 already in progress: creating tablet
I20260501 13:57:51.862234 1883 raft_consensus.cc:359] T 82d94a00960a4abeb3ddc5f1c1b2d3cd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.878955 1883 raft_consensus.cc:385] T 82d94a00960a4abeb3ddc5f1c1b2d3cd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:51.879251 1883 raft_consensus.cc:740] T 82d94a00960a4abeb3ddc5f1c1b2d3cd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:51.882277 1882 tablet_bootstrap.cc:654] T f52ea8fb8ae0427d804001aed42cd67c P 8f90b7ff170b4eadb27abc9ef866ffcf: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:51.890537 1883 consensus_queue.cc:260] T 82d94a00960a4abeb3ddc5f1c1b2d3cd P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:51.897782 1125 catalog_manager.cc:5671] T 967658b1be204c638fc34983293ef5ab P 8f90b7ff170b4eadb27abc9ef866ffcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130). New cstate: current_term: 1 leader_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } health_report { overall_health: UNKNOWN } } }
I20260501 13:57:51.898542 2582 tablet_replica.cc:333] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf: stopping tablet replica
I20260501 13:57:51.899271 2582 raft_consensus.cc:2243] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:51.899200 1125 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 967658b1be204c638fc34983293ef5ab (table default.Table_1ec94dd4afbb40d19a6f68aaefad8790 [id=f57f3fcda9444fa0a1ad062576f8d051]). Expected version 3 got 0
I20260501 13:57:51.899709 2582 raft_consensus.cc:2272] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:51.901407 2582 ts_tablet_manager.cc:1916] T 569a0240b1274d9bb830addbaf1ec343 P 8f90b7ff170b4eadb27abc9ef866ffcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:51.906323 1748 tablet_service.cc:1511] Processing CreateTablet for tablet eb147e0bf5ca4b529524590b42fe45e1 (DEFAULT_TABLE table=default.Table_c4a062dd073f477294baea6a237e0392 [id=c36d01538a954ff1acdde19a7a0769a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.909627 1883 ts_tablet_manager.cc:1434] T 82d94a00960a4abeb3ddc5f1c1b2d3cd P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20260501 13:57:51.918687 1596 tablet_service.cc:1511] Processing CreateTablet for tablet eb147e0bf5ca4b529524590b42fe45e1 (DEFAULT_TABLE table=default.Table_c4a062dd073f477294baea6a237e0392 [id=c36d01538a954ff1acdde19a7a0769a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.919946 1596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb147e0bf5ca4b529524590b42fe45e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.933209 2795 tablet_replica.cc:333] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:51.934013 2795 raft_consensus.cc:2243] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:51.934952 1757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67eb4cc6eeba4862b55366c78109b145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.939287 2795 raft_consensus.cc:2272] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:51.939422 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 301fad71fe6e4e53a36216c41f2850bf (table default.Table_9fec3571b60d4a7da2653ad41120b531 [id=70e8e791a366491eaae840dbd7565b34]). Expected version 1 got 0
I20260501 13:57:51.939610 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet b125d1a190f947a5b2ef942f04408c51 (table default.Table_536f8eeddbae410cbc4414090433de66 [id=d9b238f8c49e4842b7055439df41703d]) successfully deleted
I20260501 13:57:51.940312 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 77c59fc1fdd645af8c3017b6836c8bf4 (table default.Table_ed5738206c234f44b970d6edc256b193 [id=7829eeffca1d405ba5dbe7b4c94da149]). Expected version 1 got 0
I20260501 13:57:51.941550 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f02c141dc24f2186267c48f5ee3db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.942268 1744 tablet_service.cc:1511] Processing CreateTablet for tablet 20e288c811ee423cb13d10cabaaf5ece (DEFAULT_TABLE table=default.Table_9f3c5df0faf64dedabec83d1c7d3dd82 [id=606cf51052b443f7aacbd78885edd3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.943343 1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20e288c811ee423cb13d10cabaaf5ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.947304 1752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc485efda1414befa957251d6a7de298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.949249 2795 ts_tablet_manager.cc:1916] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:51.950950 1748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb147e0bf5ca4b529524590b42fe45e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.956656 1608 tablet_service.cc:1511] Processing CreateTablet for tablet 0760ed117df7468385cb79742b613f36 (DEFAULT_TABLE table=default.Table_f81cde0094724a1bb91d297f541f7390 [id=93b55fa04f544ba59fb99a1552b078b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.958745 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 0760ed117df7468385cb79742b613f36 (DEFAULT_TABLE table=default.Table_f81cde0094724a1bb91d297f541f7390 [id=93b55fa04f544ba59fb99a1552b078b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:51.959769 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0760ed117df7468385cb79742b613f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:51.958765 1883 tablet_bootstrap.cc:492] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:51.971371 1595 tablet_service.cc:1511] Processing CreateTablet for tablet 20e288c811ee423cb13d10cabaaf5ece (DEFAULT_TABLE table=default.Table_9f3c5df0faf64dedabec83d1c7d3dd82 [id=606cf51052b443f7aacbd78885edd3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:52.008090 1608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0760ed117df7468385cb79742b613f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:52.011112 1595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20e288c811ee423cb13d10cabaaf5ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:52.030146 1883 tablet_bootstrap.cc:654] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:52.036935 1125 catalog_manager.cc:5763] TS 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735) does not have the latest schema for tablet 301fad71fe6e4e53a36216c41f2850bf (table default.Table_9fec3571b60d4a7da2653ad41120b531 [id=70e8e791a366491eaae840dbd7565b34]). Expected version 1 got 0
I20260501 13:57:52.042435 1882 tablet_bootstrap.cc:492] T f52ea8fb8ae0427d804001aed42cd67c P 8f90b7ff170b4eadb27abc9ef866ffcf: No bootstrap required, opened a new log
I20260501 13:57:52.042882 1882 ts_tablet_manager.cc:1403] T f52ea8fb8ae0427d804001aed42cd67c P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent bootstrapping tablet: real 0.199s user 0.010s sys 0.004s
I20260501 13:57:52.043033 1591 tablet_service.cc:1558] Processing DeleteTablet for tablet 68cb7bd1a5aa4419b9c5176a287fe885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:39 UTC) from {username='slave'} at 127.0.0.1:60694
I20260501 13:57:52.045400 1882 raft_consensus.cc:359] T f52ea8fb8ae0427d804001aed42cd67c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:52.046061 1882 raft_consensus.cc:385] T f52ea8fb8ae0427d804001aed42cd67c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:52.046418 1882 raft_consensus.cc:740] T f52ea8fb8ae0427d804001aed42cd67c P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90b7ff170b4eadb27abc9ef866ffcf, State: Initialized, Role: FOLLOWER
I20260501 13:57:52.047132 1882 consensus_queue.cc:260] T f52ea8fb8ae0427d804001aed42cd67c P 8f90b7ff170b4eadb27abc9ef866ffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:52.052685 1882 ts_tablet_manager.cc:1434] T f52ea8fb8ae0427d804001aed42cd67c P 8f90b7ff170b4eadb27abc9ef866ffcf: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260501 13:57:52.056102 1752 tablet_service.cc:1558] Processing DeleteTablet for tablet 82d94a00960a4abeb3ddc5f1c1b2d3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:45 UTC) from {username='slave'} at 127.0.0.1:48970
W20260501 13:57:52.063884 1564 connection.cc:570] client connection to 127.1.16.129:38663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:57:52.064337 1564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 13:57:52.068279 1883 tablet_bootstrap.cc:492] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:52.069943 2743 raft_consensus.cc:493] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:52.070390 2743 raft_consensus.cc:515] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:52.071272 1117 connection.cc:570] client connection to 127.1.16.129:38663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:57:52.071614 1117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 2 similar messages]
I20260501 13:57:52.072464 2778 tablet.cc:1724] T 406e1af8b7d9488f88697206daa87c05 P 8f90b7ff170b4eadb27abc9ef866ffcf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:57:52.073441 2743 leader_election.cc:290] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859)
I20260501 13:57:52.075631 1883 ts_tablet_manager.cc:1403] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.117s user 0.010s sys 0.000s
I20260501 13:57:52.076539 2816 tablet.cc:1724] T 406e1af8b7d9488f88697206daa87c05 P 0ed1f9a545474e76a661c32bf11a7fb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 13:57:52.078069 1563 leader_election.cc:343] T 68cb7bd1a5aa4419b9c5176a287fe885 P 8f90b7ff170b4eadb27abc9ef866ffcf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): Not found: Tablet not found: 68cb7bd1a5aa4419b9c5176a287fe885
W20260501 13:57:52.084779 1717 connection.cc:570] client connection to 127.1.16.129:38663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:57:52.085254 1717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 13:57:52.095256 1882 tablet_bootstrap.cc:492] T 8dbb619863dd405ab20ead6bbebf4d69 P 8f90b7ff170b4eadb27abc9ef866ffcf: Bootstrap starting.
W20260501 13:57:52.096601 1116 connection.cc:570] server connection from 127.1.16.129:53385 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 13:57:52.105297 2795 ts_tablet_manager.cc:1929] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260501 13:57:52.107151 1565 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260501 13:57:52.108980 1883 raft_consensus.cc:359] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:52.109984 1090 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu with pid 1542
I20260501 13:57:52.110327 1883 raft_consensus.cc:385] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:52.113837 1883 raft_consensus.cc:740] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:52.118768 1883 consensus_queue.cc:260] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:52.120219 1117 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 48614.2us
I20260501 13:57:52.126763 1883 ts_tablet_manager.cc:1434] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.051s user 0.006s sys 0.000s
I20260501 13:57:52.127724 1883 tablet_bootstrap.cc:492] T 691464fc50314a498a5399601b43c733 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:52.116788 2795 log.cc:1199] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/ef622e34d46a4b19a59e916d22c2b18b
I20260501 13:57:52.129047 2795 ts_tablet_manager.cc:1950] T ef622e34d46a4b19a59e916d22c2b18b P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:52.132047 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet ef622e34d46a4b19a59e916d22c2b18b (table default.Table_6fe907773720414d8f8301eb3839c456 [id=498581a9af3c4ca6a97a769fa38c53c0]) successfully deleted
W20260501 13:57:52.132781 1116 catalog_manager.cc:4969] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): delete failed for tablet 8dbb619863dd405ab20ead6bbebf4d69 because the tablet was not found. No further retry: Not found: Tablet not found: 8dbb619863dd405ab20ead6bbebf4d69
I20260501 13:57:52.133627 2755 raft_consensus.cc:493] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:52.134086 2755 raft_consensus.cc:515] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:52.135247 1717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 50033.1us
I20260501 13:57:52.136557 2755 leader_election.cc:290] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
I20260501 13:57:52.137015 2755 raft_consensus.cc:493] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:52.137406 2755 raft_consensus.cc:515] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:52.139784 2755 leader_election.cc:290] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:52.151449 2795 tablet_replica.cc:333] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:52.152396 2795 raft_consensus.cc:2243] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:52.152921 2795 raft_consensus.cc:2272] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:52.156653 2831 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f04a112d3be549d98b4ee77dbefacb7b)
I20260501 13:57:52.157357 2831 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04a112d3be549d98b4ee77dbefacb7b" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 45953 } } peers { permanent_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 45897 } } peers { permanent_uuid: "50c0932256e74399a537670bb424a1b7" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38815 } }
I20260501 13:57:52.160614 1216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 396 } } ignore_live_leader: false dest_uuid: "d18ed9dc598e4ccaaf8633eaa45a2f51" is_pre_election: true
I20260501 13:57:52.160650 1140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c0932256e74399a537670bb424a1b7" candidate_term: 4 candidate_status { last_received { term: 3 index: 396 } } ignore_live_leader: false dest_uuid: "f04a112d3be549d98b4ee77dbefacb7b" is_pre_election: true
I20260501 13:57:52.162240 2451 leader_election.cc:304] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50c0932256e74399a537670bb424a1b7; no voters: d18ed9dc598e4ccaaf8633eaa45a2f51, f04a112d3be549d98b4ee77dbefacb7b
I20260501 13:57:52.179054 2795 ts_tablet_manager.cc:1916] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:57:52.184270 1117 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.16.129:38663, user_credentials={real_user=slave}} blocked reactor thread for 61803.7us
I20260501 13:57:52.189038 1883 tablet_bootstrap.cc:654] T 691464fc50314a498a5399601b43c733 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:52.181252 2833 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260501 13:57:52.192047 2831 leader_election.cc:290] T 00000000000000000000000000000000 P 50c0932256e74399a537670bb424a1b7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f04a112d3be549d98b4ee77dbefacb7b (127.1.16.190:45953), d18ed9dc598e4ccaaf8633eaa45a2f51 (127.1.16.189:45897)
W20260501 13:57:52.198385 1717 leader_election.cc:336] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.198961 1717 leader_election.cc:304] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:52.199872 1717 consensus_peers.cc:597] T a4265d9ab3734a8b97d745dc02c6b845 P 0ed1f9a545474e76a661c32bf11a7fb2 -> Peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Couldn't send request to peer 993b9ba18d8547f8af43167e727a6a24. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260501 13:57:52.200289 1717 leader_election.cc:336] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.200407 2832 raft_consensus.cc:2749] T 0946834fac5f48a6a9976c21a45e0044 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:52.200763 1717 leader_election.cc:304] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:52.201372 2832 raft_consensus.cc:2749] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:52.201560 1717 leader_election.cc:336] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.208887 1717 leader_election.cc:304] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:52.220675 1717 raft_consensus.cc:2686] T daca20eeafb141f2bf8b4233e0c8ea3b P 0ed1f9a545474e76a661c32bf11a7fb2: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260501 13:57:52.227756 1117 catalog_manager.cc:4729] TS 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): CreateTablet RPC failed for tablet faf8205072b14aceaa95658fd3093754: Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
W20260501 13:57:52.229452 1717 leader_election.cc:336] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.245576 1717 leader_election.cc:304] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:52.232342 2795 ts_tablet_manager.cc:1929] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:52.246236 2795 log.cc:1199] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/1c630c25ec5c46539936ec545ff4d292
I20260501 13:57:52.246591 2832 raft_consensus.cc:2749] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:52.246929 2795 ts_tablet_manager.cc:1950] T 1c630c25ec5c46539936ec545ff4d292 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
W20260501 13:57:52.247457 1717 leader_election.cc:336] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.252211 1717 leader_election.cc:304] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:52.248677 2795 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 371) took 1011 ms. Trace:
I20260501 13:57:52.218214 1883 tablet_bootstrap.cc:492] T 691464fc50314a498a5399601b43c733 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:52.253376 2832 raft_consensus.cc:2749] T 4f7db275918c41139cf9bf44bba001a7 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:52.253638 1883 ts_tablet_manager.cc:1403] T 691464fc50314a498a5399601b43c733 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.126s user 0.008s sys 0.003s
W20260501 13:57:52.254317 1717 leader_election.cc:336] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.253383 2795 rpcz_store.cc:276] 0501 13:57:51.237480 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:51.257488 (+ 20008us) service_pool.cc:224] Handling call
0501 13:57:52.151265 (+893777us) ts_tablet_manager.cc:1151] Deleting tablet 1c630c25ec5c46539936ec545ff4d292
0501 13:57:52.151314 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:52.192045 (+ 40731us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:52.232218 (+ 40173us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:52.248623 (+ 16405us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":893164}
I20260501 13:57:52.263309 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 1c630c25ec5c46539936ec545ff4d292 (table default.Table_220e55459a0a41c69abf3fd74bca7bdd [id=67dd2e70eed74686b327b62e2cfae1a0]) successfully deleted
I20260501 13:57:52.264158 2795 tablet_replica.cc:333] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:52.264943 2795 raft_consensus.cc:2243] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:52.256815 1883 raft_consensus.cc:359] T 691464fc50314a498a5399601b43c733 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:52.268185 2795 raft_consensus.cc:2272] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:52.268456 1883 raft_consensus.cc:385] T 691464fc50314a498a5399601b43c733 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:52.269220 1883 raft_consensus.cc:740] T 691464fc50314a498a5399601b43c733 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:52.263690 1717 leader_election.cc:304] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:52.270493 2832 raft_consensus.cc:2749] T 5a7eecc7345949cc81ad73e2d0370c9c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:52.269994 1883 consensus_queue.cc:260] T 691464fc50314a498a5399601b43c733 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
W20260501 13:57:52.271198 1717 leader_election.cc:336] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.272480 1717 leader_election.cc:304] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:52.273053 2832 raft_consensus.cc:2749] T 31c73582788648aab55d678ad6e4fc76 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:57:52.273785 1717 leader_election.cc:336] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.274122 1717 leader_election.cc:304] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:52.275372 2832 raft_consensus.cc:2749] T 1042bdf58c1f488daeaec284eb3319d5 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:52.275570 1128 catalog_manager.cc:6216] T 00000000000000000000000000000000 P f04a112d3be549d98b4ee77dbefacb7b: default.Table_759f3009542147de94c73784e1698246 [id=0f7c81f8d9a64f17921007cc8f9f3ba5] alter complete (version 1)
W20260501 13:57:52.276166 1717 leader_election.cc:336] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.276512 1717 leader_election.cc:304] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:52.280670 1883 ts_tablet_manager.cc:1434] T 691464fc50314a498a5399601b43c733 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20260501 13:57:52.280966 2795 ts_tablet_manager.cc:1916] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:52.289574 2832 raft_consensus.cc:2749] T 77c59fc1fdd645af8c3017b6836c8bf4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:52.295070 1883 tablet_bootstrap.cc:492] T 0e2953dd57af41ea98d87d299e1a5f12 P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:52.304154 1883 tablet_bootstrap.cc:654] T 0e2953dd57af41ea98d87d299e1a5f12 P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
W20260501 13:57:52.326874 1717 leader_election.cc:336] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
W20260501 13:57:52.332926 1717 leader_election.cc:336] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.360809 2832 raft_consensus.cc:493] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:52.361304 2832 raft_consensus.cc:515] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:52.363132 2832 leader_election.cc:290] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735), 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663)
I20260501 13:57:52.365172 1883 tablet_bootstrap.cc:492] T 0e2953dd57af41ea98d87d299e1a5f12 P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:52.365625 1883 ts_tablet_manager.cc:1403] T 0e2953dd57af41ea98d87d299e1a5f12 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.006s
W20260501 13:57:52.368064 1717 leader_election.cc:336] T f57755ff94ce4e4481251c988eb337e5 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.368878 1883 raft_consensus.cc:359] T 0e2953dd57af41ea98d87d299e1a5f12 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:52.386672 1883 raft_consensus.cc:385] T 0e2953dd57af41ea98d87d299e1a5f12 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:52.387084 1883 raft_consensus.cc:740] T 0e2953dd57af41ea98d87d299e1a5f12 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:52.387797 1883 consensus_queue.cc:260] T 0e2953dd57af41ea98d87d299e1a5f12 P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:52.373528 2795 ts_tablet_manager.cc:1929] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:52.392055 2795 log.cc:1199] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/1bb7fdcad3a54317b43bfd58f08e89d0
I20260501 13:57:52.392757 2795 ts_tablet_manager.cc:1950] T 1bb7fdcad3a54317b43bfd58f08e89d0 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
I20260501 13:57:52.394834 1883 ts_tablet_manager.cc:1434] T 0e2953dd57af41ea98d87d299e1a5f12 P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20260501 13:57:52.401455 2795 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48970 (request call id 380) took 1032 ms. Trace:
I20260501 13:57:52.410290 2795 rpcz_store.cc:276] 0501 13:57:51.368993 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:57:51.372891 (+ 3898us) service_pool.cc:224] Handling call
0501 13:57:52.263994 (+891103us) ts_tablet_manager.cc:1151] Deleting tablet 1bb7fdcad3a54317b43bfd58f08e89d0
0501 13:57:52.264047 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:57:52.327031 (+ 62984us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:52.373405 (+ 46374us) tablet_metadata.cc:665] Metadata flushed
0501 13:57:52.401396 (+ 27991us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":890552}
I20260501 13:57:52.407186 1883 tablet_bootstrap.cc:492] T 3223417aa5594ef7942c6d90bdc8711c P 0ed1f9a545474e76a661c32bf11a7fb2: Bootstrap starting.
I20260501 13:57:52.414692 1116 catalog_manager.cc:5002] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859): tablet 1bb7fdcad3a54317b43bfd58f08e89d0 (table default.Table_14a392c71f144eb0a4e0d5ac408e98e9 [id=a978daa811864093b075583f4e6bc4a1]) successfully deleted
I20260501 13:57:52.415314 2832 raft_consensus.cc:493] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:57:52.415758 2832 raft_consensus.cc:515] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:52.417582 2832 leader_election.cc:290] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663), 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735)
W20260501 13:57:52.424243 1717 leader_election.cc:336] T d5e1ab4acf0d400b963c3c61772f6272 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 993b9ba18d8547f8af43167e727a6a24 (127.1.16.129:38663): Network error: Client connection negotiation failed: client connection to 127.1.16.129:38663: connect: Connection refused (error 111)
I20260501 13:57:52.438653 2795 tablet_replica.cc:333] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2: stopping tablet replica
I20260501 13:57:52.440641 2795 raft_consensus.cc:2243] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:57:52.441133 2795 raft_consensus.cc:2272] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:57:52.454830 2795 ts_tablet_manager.cc:1916] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:57:52.462975 2614 mvcc.cc:204] Tried to move back new op lower bound from 7281229263076491264 to 7281229192719417344. Current Snapshot: MvccSnapshot[applied={T|T < 7281229263076491264 or (T in {7281229263076491264})}]
I20260501 13:57:52.467870 1883 tablet_bootstrap.cc:654] T 3223417aa5594ef7942c6d90bdc8711c P 0ed1f9a545474e76a661c32bf11a7fb2: Neither blocks nor log segments found. Creating new log.
I20260501 13:57:52.475087 1744 tablet_service.cc:1558] Processing DeleteTablet for tablet 53d9a8503c8d4bebb89ea493b99bf6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:57:44 UTC) from {username='slave'} at 127.0.0.1:48970
I20260501 13:57:52.479859 1744 tablet_service.cc:1511] Processing CreateTablet for tablet b8f22056d56c4054a0ac852641d7cf32 (DEFAULT_TABLE table=default.Table_67dfb9c64e154290bd6a095e8cdab9bd [id=f3536e12b9c94d659d47179e2d4aa986]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:52.480918 1744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f22056d56c4054a0ac852641d7cf32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:52.492775 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 58f602643f47476086671ad0f38ab7d5 (DEFAULT_TABLE table=default.Table_51db366aa28a41398e299c943cf4f9f5 [id=e4af5dca745e42abbf2854d0a02c82a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:52.493819 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58f602643f47476086671ad0f38ab7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:52.496783 1748 tablet_service.cc:1511] Processing CreateTablet for tablet 971ea206e04646ceb901455c823f65cb (DEFAULT_TABLE table=default.Table_75d087387b964c72956c5a5ff030763e [id=98cc3aed0eda45ffbfbe5526ff5dd3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:52.497824 1748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 971ea206e04646ceb901455c823f65cb. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:57:52.498838 1119 connection.cc:570] server connection from 127.1.16.130:35289 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:57:52.499747 1719 connection.cc:570] client connection to 127.1.16.130:36735 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:57:52.504983 1119 connection.cc:570] client connection to 127.1.16.130:36735 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 13:57:52.505366 1760 tablet_service.cc:1511] Processing CreateTablet for tablet 04a375d9f28b4ad2a85381d0ec6e4215 (DEFAULT_TABLE table=default.Table_86557c4c83aa4a56b92a6963d66ca04a [id=bd3abd4c33f040779fd3b296033a6d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:52.506299 1760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04a375d9f28b4ad2a85381d0ec6e4215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:52.515424 1763 tablet_service.cc:1511] Processing CreateTablet for tablet 40670b0208e942e08c9a6fee1b528524 (DEFAULT_TABLE table=default.Table_c3573ab598cc4732af3fe89825d4e51b [id=7c58c2a469194a0988efab1fec47cddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:57:52.519297 1763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40670b0208e942e08c9a6fee1b528524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:57:52.521103 1883 tablet_bootstrap.cc:492] T 3223417aa5594ef7942c6d90bdc8711c P 0ed1f9a545474e76a661c32bf11a7fb2: No bootstrap required, opened a new log
I20260501 13:57:52.524511 1883 ts_tablet_manager.cc:1403] T 3223417aa5594ef7942c6d90bdc8711c P 0ed1f9a545474e76a661c32bf11a7fb2: Time spent bootstrapping tablet: real 0.117s user 0.010s sys 0.008s
I20260501 13:57:52.526930 1883 raft_consensus.cc:359] T 3223417aa5594ef7942c6d90bdc8711c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:52.530584 1883 raft_consensus.cc:385] T 3223417aa5594ef7942c6d90bdc8711c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:57:52.530927 1883 raft_consensus.cc:740] T 3223417aa5594ef7942c6d90bdc8711c P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1f9a545474e76a661c32bf11a7fb2, State: Initialized, Role: FOLLOWER
I20260501 13:57:52.517623 2795 ts_tablet_manager.cc:1929] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:57:52.531642 2795 log.cc:1199] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/3b0ed592fd3642c194bb946c8c6003b4
W20260501 13:57:52.532187 2834 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 13:57:52.502395 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.1.16.130:36735 (local address 127.1.16.131:34153)
0501 13:57:52.503335 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0501 13:57:52.503380 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0501 13:57:52.503799 (+ 419us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 13:57:52.531148 (+ 27349us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.16.130:36735: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":349}
I20260501 13:57:52.532354 2795 ts_tablet_manager.cc:1950] T 3b0ed592fd3642c194bb946c8c6003b4 P 0ed1f9a545474e76a661c32bf11a7fb2: Deleting consensus metadata
W20260501 13:57:52.532603 1719 leader_election.cc:336] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Network error: Client connection negotiation failed: client connection to 127.1.16.130:36735: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 13:57:52.532966 1719 leader_election.cc:304] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
W20260501 13:57:52.533736 1719 leader_election.cc:336] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 8f90b7ff170b4eadb27abc9ef866ffcf (127.1.16.130:36735): Network error: Client connection negotiation failed: client connection to 127.1.16.130:36735: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:57:52.533944 2835 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 13:57:52.513379 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.1.16.130:36735 (local address 127.0.0.1:37928)
0501 13:57:52.514042 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0501 13:57:52.514088 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0501 13:57:52.514355 (+ 267us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 13:57:52.533099 (+ 18744us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.16.130:36735: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":93}
I20260501 13:57:52.533643 1883 consensus_queue.cc:260] T 3223417aa5594ef7942c6d90bdc8711c P 0ed1f9a545474e76a661c32bf11a7fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993b9ba18d8547f8af43167e727a6a24" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 38663 } } peers { permanent_uuid: "8f90b7ff170b4eadb27abc9ef866ffcf" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 36735 } } peers { permanent_uuid: "0ed1f9a545474e76a661c32bf11a7fb2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 34859 } }
I20260501 13:57:52.534132 1719 leader_election.cc:304] T 50ec3d3f3eb449759a8df52279b2e5e8 P 0ed1f9a545474e76a661c32bf11a7fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ed1f9a545474e76a661c32bf11a7fb2; no voters: 8f90b7ff170b4eadb27abc9ef866ffcf, 993b9ba18d8547f8af43167e727a6a24
I20260501 13:57:52.538636 2832 raft_consensus.cc:2749] T 53d9a8503c8d4bebb89ea493b99bf6fd P 0ed1f9a545474e76a661c32bf11a7fb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:57:52.544656 1090 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu with pid 1685
I20260501 13:57:52.591989 1128 catalog_manager.cc:5763] TS 0ed1f9a545474e76a661c32bf11a7fb2 (127.1.16.131:34859) does not have the latest schema for tablet 0e2953dd57af41ea98d87d299e1a5f12 (table default.Table_1da5624be9fd43498a32173c9025a446 [id=2fb661abb47848e2b4a7df024ecb18f3]). Expected version 1 got 0
W20260501 13:57:52.603292 1117 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet ef191926320a4e0fa8bd94e47e9b9a86 (table default.Table_13bc55336f6e4f9d8ec006df8bb8f0aa [id=e8b19350d702499a82f41a6cce14dbd5], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet ef191926320a4e0fa8bd94e47e9b9a86 consensus state has no leader
W20260501 13:57:52.657152 1116 connection.cc:570] client connection to 127.1.16.131:34859 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 13:57:52.658663 1090 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu with pid 1099
I20260501 13:57:52.727227 1090 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu with pid 1169
I20260501 13:57:52.780639 1090 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu with pid 2028
2026-05-01T13:57:52Z chronyd exiting
I20260501 13:57:52.829941 1090 master-stress-test.cc:585] Tables created: 245
I20260501 13:57:52.830169 1090 master-stress-test.cc:586] Tables altered: 123
I20260501 13:57:52.830313 1090 master-stress-test.cc:587] Tables deleted: 83
I20260501 13:57:52.830440 1090 master-stress-test.cc:588] Tablets replaced: 25
I20260501 13:57:52.830612 1090 master-stress-test.cc:589] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (39860 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
2026-05-01T13:57:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T13:57:52Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task2LI5eU/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task2LI5eU/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/security.properties
2026-05-01 13:58:00: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task2LI5eU/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task2LI5eU/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260501 13:58:07.891345 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.190:38561
--webserver_interface=127.1.16.190
--webserver_port=0
--builtin_ntp_servers=127.1.16.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--hive_metastore_uris=thrift://127.0.0.1:45913
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260501 13:58:08.192554 3121 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:58:08.193142 3121 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:58:08.193569 3121 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 13:58:08.226496 3121 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260501 13:58:08.226778 3121 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260501 13:58:08.227020 3121 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 13:58:08.227254 3121 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:58:08.227485 3121 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 13:58:08.227710 3121 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 13:58:08.268168 3121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45913
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.190:38561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.16.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:58:08.269500 3121 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:58:08.271131 3121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 13:58:08.281720 3127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:09.684865 3126 debug-util.cc:398] Leaking SignalData structure 0x7b080002ecc0 after lost signal to thread 3121
W20260501 13:58:09.832329 3121 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.550s user 0.562s sys 0.983s
W20260501 13:58:08.283020 3128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:09.832798 3121 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.551s user 0.562s sys 0.983s
W20260501 13:58:09.834625 3130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:09.837524 3129 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1550 milliseconds
I20260501 13:58:09.837566 3121 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260501 13:58:09.838768 3121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:58:09.841156 3121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:58:09.842532 3121 hybrid_clock.cc:648] HybridClock initialized: now 1777643889842467 us; error 71 us; skew 500 ppm
I20260501 13:58:09.843307 3121 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:58:09.849309 3121 webserver.cc:492] Webserver started at http://127.1.16.190:41345/ using document root <none> and password file <none>
I20260501 13:58:09.850274 3121 fs_manager.cc:362] Metadata directory not provided
I20260501 13:58:09.850540 3121 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:58:09.850989 3121 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:58:09.855438 3121 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/data/instance:
uuid: "0a1d2cd221924677a3ef81b081e5e016"
format_stamp: "Formatted at 2026-05-01 13:58:09 on dist-test-slave-nqxd"
I20260501 13:58:09.856626 3121 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/wal/instance:
uuid: "0a1d2cd221924677a3ef81b081e5e016"
format_stamp: "Formatted at 2026-05-01 13:58:09 on dist-test-slave-nqxd"
I20260501 13:58:09.864766 3121 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.002s
I20260501 13:58:09.871748 3137 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:09.872836 3121 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260501 13:58:09.873162 3121 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/wal
uuid: "0a1d2cd221924677a3ef81b081e5e016"
format_stamp: "Formatted at 2026-05-01 13:58:09 on dist-test-slave-nqxd"
I20260501 13:58:09.873473 3121 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:58:09.930162 3121 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:58:09.931684 3121 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:58:09.932125 3121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:58:10.000782 3121 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.190:38561
I20260501 13:58:10.000846 3188 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.190:38561 every 8 connection(s)
I20260501 13:58:10.003451 3121 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/data/info.pb
I20260501 13:58:10.009107 3189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:10.010916 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 3121
I20260501 13:58:10.011364 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-0/wal/instance
I20260501 13:58:10.013794 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.189:44255
--webserver_interface=127.1.16.189
--webserver_port=0
--builtin_ntp_servers=127.1.16.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--hive_metastore_uris=thrift://127.0.0.1:45913
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260501 13:58:10.019984 3189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:58:10.052371 3189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } has no permanent_uuid. Determining permanent_uuid...
W20260501 13:58:10.055075 3140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
W20260501 13:58:10.059478 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:44255: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
I20260501 13:58:10.109098 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } attempt: 1
W20260501 13:58:10.112969 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:44255: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
I20260501 13:58:10.181501 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } attempt: 2
W20260501 13:58:10.185760 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:44255: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
I20260501 13:58:10.277298 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } attempt: 3
W20260501 13:58:10.281231 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:44255: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
W20260501 13:58:10.329313 3190 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:58:10.329880 3190 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:58:10.330274 3190 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 13:58:10.374950 3190 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260501 13:58:10.375217 3190 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260501 13:58:10.375422 3190 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 13:58:10.375622 3190 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:58:10.375820 3190 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 13:58:10.376025 3190 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 13:58:10.413828 3190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45913
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.189:44255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.16.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:58:10.415275 3190 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:58:10.416922 3190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260501 13:58:10.424738 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } attempt: 4
W20260501 13:58:10.427520 3199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:10.430717 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:44255: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
I20260501 13:58:10.730232 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } attempt: 5
W20260501 13:58:10.736635 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:44255: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
I20260501 13:58:11.284158 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } attempt: 6
W20260501 13:58:11.319609 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.189:44255: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
W20260501 13:58:10.429735 3200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:11.696880 3202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:11.698904 3201 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1270 milliseconds
W20260501 13:58:11.699712 3190 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.272s user 0.411s sys 0.840s
W20260501 13:58:11.699961 3190 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.272s user 0.411s sys 0.840s
I20260501 13:58:11.700145 3190 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260501 13:58:11.701149 3190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:58:11.703807 3190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:58:11.705224 3190 hybrid_clock.cc:648] HybridClock initialized: now 1777643891705187 us; error 45 us; skew 500 ppm
I20260501 13:58:11.706205 3190 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:58:11.715404 3190 webserver.cc:492] Webserver started at http://127.1.16.189:36257/ using document root <none> and password file <none>
I20260501 13:58:11.716610 3190 fs_manager.cc:362] Metadata directory not provided
I20260501 13:58:11.716895 3190 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:58:11.717353 3190 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:58:11.721935 3190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data/instance:
uuid: "15dd70d7c5494c3c911f0a780dc9a2af"
format_stamp: "Formatted at 2026-05-01 13:58:11 on dist-test-slave-nqxd"
I20260501 13:58:11.723371 3190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal/instance:
uuid: "15dd70d7c5494c3c911f0a780dc9a2af"
format_stamp: "Formatted at 2026-05-01 13:58:11 on dist-test-slave-nqxd"
I20260501 13:58:11.731446 3190 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260501 13:58:11.737731 3211 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:11.739050 3190 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260501 13:58:11.739413 3190 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal
uuid: "15dd70d7c5494c3c911f0a780dc9a2af"
format_stamp: "Formatted at 2026-05-01 13:58:11 on dist-test-slave-nqxd"
I20260501 13:58:11.739763 3190 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:58:11.804921 3190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:58:11.806576 3190 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:58:11.807039 3190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:58:11.876979 3190 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.189:44255
I20260501 13:58:11.877058 3262 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.189:44255 every 8 connection(s)
I20260501 13:58:11.879684 3190 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data/info.pb
I20260501 13:58:11.882884 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 3190
I20260501 13:58:11.883273 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal/instance
I20260501 13:58:11.885299 3263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:11.885614 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.188:38253
--webserver_interface=127.1.16.188
--webserver_port=0
--builtin_ntp_servers=127.1.16.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--hive_metastore_uris=thrift://127.0.0.1:45913
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260501 13:58:11.896505 3263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:58:11.918028 3263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:58:11.931917 3263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } has no permanent_uuid. Determining permanent_uuid...
W20260501 13:58:11.934203 3213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
W20260501 13:58:11.938360 3263 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:11.987900 3263 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 1
W20260501 13:58:11.992123 3263 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:12.060523 3263 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 2
W20260501 13:58:12.064693 3263 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:12.156136 3263 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 3
W20260501 13:58:12.160409 3263 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
W20260501 13:58:12.207324 3264 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:58:12.207911 3264 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:58:12.208315 3264 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 13:58:12.242408 3264 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260501 13:58:12.242748 3264 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260501 13:58:12.243067 3264 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 13:58:12.243314 3264 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:58:12.243552 3264 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 13:58:12.243786 3264 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 13:58:12.281090 3264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45913
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.188:38253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.16.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:58:12.282455 3264 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:58:12.284162 3264 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 13:58:12.295500 3274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:58:12.303813 3263 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 4
W20260501 13:58:12.309769 3263 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:12.380142 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } attempt: 7
I20260501 13:58:12.411955 3189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } has no permanent_uuid. Determining permanent_uuid...
W20260501 13:58:12.426016 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:12.484519 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 1
W20260501 13:58:12.490892 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:12.572425 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 2
W20260501 13:58:12.578969 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:12.609243 3263 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 5
W20260501 13:58:12.615988 3263 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:12.664505 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 3
W20260501 13:58:12.671154 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:12.811641 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 4
W20260501 13:58:12.817853 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:13.101362 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 5
W20260501 13:58:13.107899 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:13.163506 3263 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 6
W20260501 13:58:13.193996 3263 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:13.660424 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 6
W20260501 13:58:12.297698 3275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:13.677023 3276 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1378 milliseconds
W20260501 13:58:13.677345 3277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:13.678998 3189 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.1.16.188:38253: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
W20260501 13:58:13.679538 3264 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.383s user 0.469s sys 0.891s
W20260501 13:58:13.679874 3264 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.384s user 0.469s sys 0.891s
I20260501 13:58:13.680147 3264 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260501 13:58:13.683156 3264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:58:13.685745 3264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:58:13.687112 3264 hybrid_clock.cc:648] HybridClock initialized: now 1777643893687077 us; error 45 us; skew 500 ppm
I20260501 13:58:13.688047 3264 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:58:13.695776 3264 webserver.cc:492] Webserver started at http://127.1.16.188:46681/ using document root <none> and password file <none>
I20260501 13:58:13.696722 3264 fs_manager.cc:362] Metadata directory not provided
I20260501 13:58:13.696952 3264 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:58:13.697402 3264 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:58:13.701776 3264 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data/instance:
uuid: "6b0f8911dd5d431d9d8d3124a4456262"
format_stamp: "Formatted at 2026-05-01 13:58:13 on dist-test-slave-nqxd"
I20260501 13:58:13.703006 3264 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal/instance:
uuid: "6b0f8911dd5d431d9d8d3124a4456262"
format_stamp: "Formatted at 2026-05-01 13:58:13 on dist-test-slave-nqxd"
I20260501 13:58:13.711211 3264 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260501 13:58:13.717557 3291 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:13.718784 3264 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260501 13:58:13.719131 3264 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal
uuid: "6b0f8911dd5d431d9d8d3124a4456262"
format_stamp: "Formatted at 2026-05-01 13:58:13 on dist-test-slave-nqxd"
I20260501 13:58:13.719475 3264 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:58:13.802731 3264 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:58:13.804229 3264 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:58:13.804663 3264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:58:13.875764 3264 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.188:38253
I20260501 13:58:13.875852 3342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.188:38253 every 8 connection(s)
I20260501 13:58:13.878767 3264 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
I20260501 13:58:13.885843 3343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:13.888298 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 3264
I20260501 13:58:13.888885 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal/instance
I20260501 13:58:13.899408 3343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:58:13.922307 3343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:58:13.932623 3343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } has no permanent_uuid. Determining permanent_uuid...
I20260501 13:58:13.948413 3343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Bootstrap starting.
I20260501 13:58:13.955441 3343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:13.957201 3343 log.cc:826] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Log is configured to *not* fsync() on all Append() calls
I20260501 13:58:13.961547 3343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: No bootstrap required, opened a new log
I20260501 13:58:13.981837 3343 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:13.982497 3343 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:13.982738 3343 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0f8911dd5d431d9d8d3124a4456262, State: Initialized, Role: FOLLOWER
I20260501 13:58:13.983426 3343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:13.985569 3351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:13.986394 3351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:13.987211 3343 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: configured and running, proceeding with master startup.
W20260501 13:58:14.035012 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.035475 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.013640 3264 master_runner.cc:186] Error getting master registration for 127.1.16.190:38561: OK, instance_id { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" instance_seqno: 1777643889982512 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260501 13:58:14.042804 3264 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260501 13:58:14.049631 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.050113 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.059849 3297 tablet.cc:2404] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 13:58:14.064978 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.065343 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.079695 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.080018 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.092968 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.093243 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.106212 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.106539 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.119545 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.119915 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.119963 3351 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:14.120476 3351 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:14.123090 3351 leader_election.cc:290] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255)
W20260501 13:58:14.134488 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.134837 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.149504 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.149855 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.164129 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.164500 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.178709 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.179047 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.193238 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.193563 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.207824 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.208184 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.222282 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.222651 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.235852 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.236119 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.251041 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.251386 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.254537 3263 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 7
W20260501 13:58:14.265558 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.265954 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.277181 3263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Bootstrap starting.
W20260501 13:58:14.280507 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.280856 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.285029 3263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:14.287449 3263 log.cc:826] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Log is configured to *not* fsync() on all Append() calls
I20260501 13:58:14.293056 3263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: No bootstrap required, opened a new log
W20260501 13:58:14.295006 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.295333 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.309904 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.310256 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.321745 3263 raft_consensus.cc:359] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:14.322777 3263 raft_consensus.cc:385] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:14.323156 3263 raft_consensus.cc:740] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15dd70d7c5494c3c911f0a780dc9a2af, State: Initialized, Role: FOLLOWER
W20260501 13:58:14.324525 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.324858 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.324288 3263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:14.327787 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:14.328737 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:14.330286 3263 sys_catalog.cc:565] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: configured and running, proceeding with master startup.
I20260501 13:58:14.330760 3238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:14.331620 3238 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b0f8911dd5d431d9d8d3124a4456262 in term 0.
I20260501 13:58:14.334583 3294 leader_election.cc:304] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262; no voters:
I20260501 13:58:14.335609 3351 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:14.335991 3351 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:14.336290 3351 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:14.342621 3351 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:14.353653 3238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af"
W20260501 13:58:14.357363 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.357703 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.354274 3238 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:58:14.371927 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.372264 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.376680 3238 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b0f8911dd5d431d9d8d3124a4456262 in term 1.
I20260501 13:58:14.374667 3351 leader_election.cc:290] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 1 election: Requested vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255)
I20260501 13:58:14.378609 3294 leader_election.cc:304] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262; no voters:
I20260501 13:58:14.380575 3351 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:14.383248 3351 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 1 LEADER]: Becoming Leader. State: Replica: 6b0f8911dd5d431d9d8d3124a4456262, State: Running, Role: LEADER
I20260501 13:58:14.384315 3351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
W20260501 13:58:14.403529 3365 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.403909 3365 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.406179 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0f8911dd5d431d9d8d3124a4456262. Latest consensus state: current_term: 1 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:14.407130 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: LEADER
I20260501 13:58:14.413336 3380 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 13:58:14.421663 3380 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 13:58:14.447780 3238 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Refusing update from remote peer 6b0f8911dd5d431d9d8d3124a4456262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:58:14.449560 3376 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:58:14.511749 3389 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.512174 3389 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.452311 3190 master_runner.cc:186] Error getting master registration for 127.1.16.190:38561: OK, instance_id { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" instance_seqno: 1777643889982512 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260501 13:58:14.538525 3190 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260501 13:58:14.545148 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0f8911dd5d431d9d8d3124a4456262. Latest consensus state: current_term: 1 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:14.545862 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:14.550879 3385 mvcc.cc:204] Tried to move back new op lower bound from 7281229391627976704 to 7281229391408791552. Current Snapshot: MvccSnapshot[applied={T|T < 7281229391627976704}]
W20260501 13:58:14.550336 3389 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 13:58:14.554527 3389 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.556569 3351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:14.557263 3351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: LEADER
I20260501 13:58:14.560613 3380 catalog_manager.cc:1357] Generated new cluster ID: 4281e294e62c41c781e2041e1b24de7c
I20260501 13:58:14.560967 3380 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 13:58:14.565114 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:14.565806 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:14.583045 3389 catalog_manager.cc:1269] Loaded cluster ID: 4281e294e62c41c781e2041e1b24de7c
I20260501 13:58:14.583389 3389 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: loading cluster ID for follower catalog manager: success
W20260501 13:58:14.590397 3389 catalog_manager.cc:1586] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260501 13:58:14.590802 3389 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.606774 3389 catalog_manager.cc:1586] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260501 13:58:14.607148 3389 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20260501 13:58:14.635525 3389 catalog_manager.cc:1586] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260501 13:58:14.636440 3389 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.644063 3380 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 13:58:14.646219 3380 catalog_manager.cc:1514] Loading token signing keys...
W20260501 13:58:14.648291 3293 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.1.16.190:38561 timed out after 0.524s (SENT)
I20260501 13:58:14.678675 3389 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: acquiring CA information for follower catalog manager: success
W20260501 13:58:14.685083 3389 catalog_manager.cc:883] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260501 13:58:14.695655 3380 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Generated new TSK 0
I20260501 13:58:14.696841 3380 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 13:58:14.701320 3380 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260501 13:58:14.705605 3380 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260501 13:58:14.712405 3189 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } attempt: 7
I20260501 13:58:14.712587 3389 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 13:58:14.743480 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.16.129:0
--local_ip_for_outbound_sockets=127.1.16.129
--webserver_interface=127.1.16.129
--webserver_port=0
--tserver_master_addrs=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--builtin_ntp_servers=127.1.16.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260501 13:58:14.759183 3189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: Bootstrap starting.
I20260501 13:58:14.766790 3189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:14.788935 3189 log.cc:826] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: Log is configured to *not* fsync() on all Append() calls
I20260501 13:58:14.797626 3189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: No bootstrap required, opened a new log
I20260501 13:58:14.838048 3189 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:14.839049 3189 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:14.839501 3189 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1d2cd221924677a3ef81b081e5e016, State: Initialized, Role: FOLLOWER
I20260501 13:58:14.840581 3189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:14.845245 3189 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 13:58:14.845711 3163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016"
I20260501 13:58:14.846424 3163 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:14.851007 3161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:14.852522 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
W20260501 13:58:14.853478 3164 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54926 (request call id 2) took 729 ms (client timeout 524 ms). Trace:
W20260501 13:58:14.853820 3164 rpcz_store.cc:269] 0501 13:58:14.124024 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:14.124138 (+ 114us) service_pool.cc:224] Handling call
0501 13:58:14.853385 (+729247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260501 13:58:14.859293 3163 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b0f8911dd5d431d9d8d3124a4456262 in term 1.
I20260501 13:58:14.860818 3162 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Refusing update from remote peer 6b0f8911dd5d431d9d8d3124a4456262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:58:14.869416 3376 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:14.844648 3399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:14.880519 3399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:14.919385 3404 mvcc.cc:204] Tried to move back new op lower bound from 7281229391627976704 to 7281229391408791552. Current Snapshot: MvccSnapshot[applied={T|T < 7281229391627976704}]
I20260501 13:58:14.949715 3399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0f8911dd5d431d9d8d3124a4456262. Latest consensus state: current_term: 1 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:14.950436 3399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:14.951376 3403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:14.953279 3403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:15.007407 3419 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260501 13:58:15.043071 3419 catalog_manager.cc:1269] Loaded cluster ID: 4281e294e62c41c781e2041e1b24de7c
I20260501 13:58:15.048084 3419 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: loading cluster ID for follower catalog manager: success
I20260501 13:58:15.071851 3419 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: acquiring CA information for follower catalog manager: success
I20260501 13:58:15.101951 3419 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 13:58:15.202821 3421 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:15.203408 3421 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:15.207441 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:15.210506 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:15.212033 3421 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:15.212445 3213 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:15.217542 3421 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:15.323979 3426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:15.332274 3426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: LEADER
W20260501 13:58:15.352633 3397 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:58:15.353425 3397 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:58:15.354431 3397 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 13:58:15.414953 3397 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:58:15.416608 3397 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.16.129
I20260501 13:58:15.486276 3397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.16.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.16.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.16.129
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:58:15.488565 3397 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:58:15.491078 3397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 13:58:15.535148 3430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:58:16.323683 3436 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:16.324245 3436 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:16.331713 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:16.331717 3238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:16.333683 3139 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016; no voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:16.347056 3436 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:16.350690 3436 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:16.948526 3438 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:16.949306 3438 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:16.952188 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:16.954874 3438 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:16.955571 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:16.957185 3213 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:16.958485 3438 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:17.743342 3439 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:17.743985 3439 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:17.749019 3439 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:17.749747 3238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:17.751442 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:17.753451 3140 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016; no voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:17.762981 3439 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:58:15.543097 3431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:16.913704 3429 debug-util.cc:398] Leaking SignalData structure 0x7b0800028ca0 after lost signal to thread 3397
W20260501 13:58:17.963665 3429 kernel_stack_watchdog.cc:198] Thread 3397 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 373ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260501 13:58:17.964457 3397 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.426s user 0.403s sys 0.726s
W20260501 13:58:17.964871 3397 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.426s user 0.403s sys 0.726s
W20260501 13:58:17.971416 3432 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 2433 milliseconds
W20260501 13:58:17.972172 3434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:58:17.972702 3397 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260501 13:58:17.974527 3397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:58:17.977222 3397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:58:17.978718 3397 hybrid_clock.cc:648] HybridClock initialized: now 1777643897978645 us; error 89 us; skew 500 ppm
I20260501 13:58:17.979863 3397 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:58:17.994807 3397 webserver.cc:492] Webserver started at http://127.1.16.129:43875/ using document root <none> and password file <none>
I20260501 13:58:17.996259 3397 fs_manager.cc:362] Metadata directory not provided
I20260501 13:58:17.996558 3397 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:58:17.997155 3397 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:58:17.999123 3443 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:17.999701 3443 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:18.001819 3443 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:18.003284 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:18.004545 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:18.005414 3397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/data/instance:
uuid: "9154033cea374a10a5225c53b8bcf122"
format_stamp: "Formatted at 2026-05-01 13:58:17 on dist-test-slave-nqxd"
I20260501 13:58:18.006909 3213 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:18.007133 3397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/instance:
uuid: "9154033cea374a10a5225c53b8bcf122"
format_stamp: "Formatted at 2026-05-01 13:58:17 on dist-test-slave-nqxd"
I20260501 13:58:18.007764 3443 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:18.024515 3397 fs_manager.cc:696] Time spent creating directory manager: real 0.017s user 0.006s sys 0.008s
I20260501 13:58:18.039362 3446 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:18.040877 3397 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.004s sys 0.007s
I20260501 13:58:18.041299 3397 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal
uuid: "9154033cea374a10a5225c53b8bcf122"
format_stamp: "Formatted at 2026-05-01 13:58:17 on dist-test-slave-nqxd"
I20260501 13:58:18.041744 3397 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:58:18.119573 3397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:58:18.121604 3397 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:58:18.122200 3397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:58:18.125914 3397 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 13:58:18.132066 3397 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 13:58:18.132376 3397 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260501 13:58:18.132695 3397 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 13:58:18.132920 3397 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:18.308300 3556 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:18.309299 3556 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:18.312808 3238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:18.314632 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:18.316344 3556 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:18.316324 3139 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016; no voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:18.317366 3556 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:18.318218 3397 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.129:44335
I20260501 13:58:18.318396 3560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.129:44335 every 8 connection(s)
I20260501 13:58:18.320979 3397 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/data/info.pb
I20260501 13:58:18.331700 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 3397
I20260501 13:58:18.332252 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/instance
I20260501 13:58:18.359819 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.16.130:0
--local_ip_for_outbound_sockets=127.1.16.130
--webserver_interface=127.1.16.130
--webserver_port=0
--tserver_master_addrs=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--builtin_ntp_servers=127.1.16.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260501 13:58:18.393293 3562 heartbeater.cc:344] Connected to a master server at 127.1.16.189:44255
I20260501 13:58:18.393859 3562 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:18.394405 3565 heartbeater.cc:344] Connected to a master server at 127.1.16.190:38561
I20260501 13:58:18.394783 3565 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:18.395315 3562 heartbeater.cc:507] Master 127.1.16.189:44255 requested a full tablet report, sending...
I20260501 13:58:18.395788 3565 heartbeater.cc:507] Master 127.1.16.190:38561 requested a full tablet report, sending...
I20260501 13:58:18.399394 3154 ts_manager.cc:194] Registered new tserver with Master: 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:18.400516 3561 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38253
I20260501 13:58:18.400876 3561 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:18.401787 3561 heartbeater.cc:507] Master 127.1.16.188:38253 requested a full tablet report, sending...
I20260501 13:58:18.405052 3228 ts_manager.cc:194] Registered new tserver with Master: 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:18.407003 3308 ts_manager.cc:194] Registered new tserver with Master: 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:18.409749 3308 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.16.129:59175
W20260501 13:58:18.705547 3571 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:58:18.706202 3571 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:58:18.706866 3571 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 13:58:18.743247 3571 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:58:18.744172 3571 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.16.130
I20260501 13:58:18.783571 3571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.16.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.16.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.16.130
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:58:18.785000 3571 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:58:18.786741 3571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 13:58:18.800436 3577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:58:18.894369 3583 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:18.895118 3583 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:18.899220 3238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:18.900287 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:18.902170 3139 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016; no voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:18.913821 3561 heartbeater.cc:499] Master 127.1.16.188:38253 was elected leader, sending a full tablet report...
I20260501 13:58:18.924230 3584 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:18.924944 3583 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:18.927529 3585 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:18.929067 3585 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:18.932164 3585 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:18.933805 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:18.934669 3164 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15dd70d7c5494c3c911f0a780dc9a2af in term 1.
I20260501 13:58:18.936846 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:18.938760 3213 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016, 15dd70d7c5494c3c911f0a780dc9a2af; no voters:
I20260501 13:58:18.950984 3585 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260501 13:58:18.954084 3585 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:18.954641 3585 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 1 FOLLOWER]: Advancing to term 2
I20260501 13:58:18.961812 3585 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:18.974982 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016"
I20260501 13:58:18.975606 3164 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 1 FOLLOWER]: Advancing to term 2
I20260501 13:58:18.977629 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262"
I20260501 13:58:18.979794 3585 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 2 election: Requested vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:19.021092 3164 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15dd70d7c5494c3c911f0a780dc9a2af in term 2.
I20260501 13:58:19.038353 3213 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016, 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:19.039659 3585 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 2 FOLLOWER]: Leader election won for term 2
I20260501 13:58:19.040216 3585 raft_consensus.cc:697] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 2 LEADER]: Becoming Leader. State: Replica: 15dd70d7c5494c3c911f0a780dc9a2af, State: Running, Role: LEADER
I20260501 13:58:19.041239 3585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:19.054154 3164 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Rejecting Update request from peer 6b0f8911dd5d431d9d8d3124a4456262 for earlier term 1. Current term is 2. Ops: []
I20260501 13:58:19.055650 3587 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260501 13:58:19.056610 3588 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 1 LEADER]: Stepping down as leader of term 1
I20260501 13:58:19.057503 3588 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b0f8911dd5d431d9d8d3124a4456262, State: Running, Role: LEADER
I20260501 13:58:19.058225 3588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:19.059300 3588 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 1 FOLLOWER]: Advancing to term 2
I20260501 13:58:19.060293 3238 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 2 LEADER]: Rejecting Update request from peer 6b0f8911dd5d431d9d8d3124a4456262 for earlier term 1. Current term is 2. Ops: []
I20260501 13:58:19.065663 3586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15dd70d7c5494c3c911f0a780dc9a2af. Latest consensus state: current_term: 2 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:19.066504 3586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: LEADER
I20260501 13:58:19.082013 3365 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 13:58:19.082502 3590 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 13:58:19.095460 3590 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 13:58:19.099773 3590 catalog_manager.cc:1269] Loaded cluster ID: 4281e294e62c41c781e2041e1b24de7c
I20260501 13:58:19.100919 3590 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 13:58:19.106889 3590 catalog_manager.cc:1514] Loading token signing keys...
I20260501 13:58:19.112190 3590 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Loaded TSK: 0
I20260501 13:58:19.113520 3590 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 13:58:19.116796 3590 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260501 13:58:19.120576 3590 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260501 13:58:19.453517 3164 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Refusing update from remote peer 15dd70d7c5494c3c911f0a780dc9a2af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260501 13:58:19.455729 3586 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260501 13:58:19.490720 3591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15dd70d7c5494c3c911f0a780dc9a2af. Latest consensus state: current_term: 2 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:19.492466 3591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:19.512856 3591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:19.515555 3591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:19.519476 3318 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 2 FOLLOWER]: Refusing update from remote peer 15dd70d7c5494c3c911f0a780dc9a2af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260501 13:58:19.519974 3585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:19.520787 3585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: LEADER
I20260501 13:58:19.521672 3586 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260501 13:58:19.556563 3588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15dd70d7c5494c3c911f0a780dc9a2af. Latest consensus state: current_term: 2 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:19.557370 3588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:19.559333 3587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:19.559990 3587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:19.919375 3562 heartbeater.cc:499] Master 127.1.16.189:44255 was elected leader, sending a full tablet report...
I20260501 13:58:20.026252 3587 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 15dd70d7c5494c3c911f0a780dc9a2af)
I20260501 13:58:20.026793 3587 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:20.028679 3587 leader_election.cc:290] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255)
I20260501 13:58:20.030689 3163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:20.032562 3237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:20.035351 3294 leader_election.cc:304] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0f8911dd5d431d9d8d3124a4456262; no voters: 0a1d2cd221924677a3ef81b081e5e016, 15dd70d7c5494c3c911f0a780dc9a2af
I20260501 13:58:20.036697 3587 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260501 13:58:20.131093 3585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:20.132246 3585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: LEADER
W20260501 13:58:20.202502 3576 debug-util.cc:398] Leaking SignalData structure 0x7b0800028ca0 after lost signal to thread 3571
W20260501 13:58:20.407447 3576 kernel_stack_watchdog.cc:198] Thread 3571 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260501 13:58:20.407788 3571 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.606s user 0.533s sys 0.950s
W20260501 13:58:18.802250 3578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:20.408279 3571 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.606s user 0.533s sys 0.950s
W20260501 13:58:20.410220 3580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:20.412210 3579 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1610 milliseconds
I20260501 13:58:20.412302 3571 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260501 13:58:20.413610 3571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:58:20.416492 3571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:58:20.417865 3571 hybrid_clock.cc:648] HybridClock initialized: now 1777643900417822 us; error 57 us; skew 500 ppm
I20260501 13:58:20.418800 3571 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:58:20.425293 3571 webserver.cc:492] Webserver started at http://127.1.16.130:34989/ using document root <none> and password file <none>
I20260501 13:58:20.426342 3571 fs_manager.cc:362] Metadata directory not provided
I20260501 13:58:20.426591 3571 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:58:20.427031 3571 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:58:20.431447 3571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/data/instance:
uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
format_stamp: "Formatted at 2026-05-01 13:58:20 on dist-test-slave-nqxd"
I20260501 13:58:20.432735 3571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/instance:
uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
format_stamp: "Formatted at 2026-05-01 13:58:20 on dist-test-slave-nqxd"
I20260501 13:58:20.441560 3571 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s
I20260501 13:58:20.447794 3605 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:20.449023 3571 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260501 13:58:20.449404 3571 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal
uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
format_stamp: "Formatted at 2026-05-01 13:58:20 on dist-test-slave-nqxd"
I20260501 13:58:20.449735 3571 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:58:20.485468 3607 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 15dd70d7c5494c3c911f0a780dc9a2af)
I20260501 13:58:20.486109 3607 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:20.488369 3607 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:20.489668 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:20.490307 3237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:20.492074 3140 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016; no voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:20.492882 3607 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260501 13:58:20.519374 3571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:58:20.520888 3571 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:58:20.521340 3571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:58:20.524178 3571 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 13:58:20.528380 3571 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 13:58:20.528584 3571 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:20.528839 3571 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 13:58:20.528996 3571 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:20.666538 3571 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.130:45857
I20260501 13:58:20.666719 3718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.130:45857 every 8 connection(s)
I20260501 13:58:20.669162 3571 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/data/info.pb
I20260501 13:58:20.677501 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 3571
I20260501 13:58:20.678016 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/instance
I20260501 13:58:20.694589 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.16.131:0
--local_ip_for_outbound_sockets=127.1.16.131
--webserver_interface=127.1.16.131
--webserver_port=0
--tserver_master_addrs=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--builtin_ntp_servers=127.1.16.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260501 13:58:20.715613 3719 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38253
I20260501 13:58:20.716162 3719 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:20.717541 3719 heartbeater.cc:507] Master 127.1.16.188:38253 requested a full tablet report, sending...
I20260501 13:58:20.720779 3720 heartbeater.cc:344] Connected to a master server at 127.1.16.189:44255
I20260501 13:58:20.721150 3720 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:20.722052 3720 heartbeater.cc:507] Master 127.1.16.189:44255 requested a full tablet report, sending...
I20260501 13:58:20.722128 3308 ts_manager.cc:194] Registered new tserver with Master: 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:20.722368 3721 heartbeater.cc:344] Connected to a master server at 127.1.16.190:38561
I20260501 13:58:20.722759 3721 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:20.723672 3721 heartbeater.cc:507] Master 127.1.16.190:38561 requested a full tablet report, sending...
I20260501 13:58:20.724495 3228 ts_manager.cc:194] Registered new tserver with Master: 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:20.726392 3154 ts_manager.cc:194] Registered new tserver with Master: 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:20.726617 3228 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.16.130:59555
W20260501 13:58:21.021313 3729 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:58:21.021811 3729 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:58:21.022370 3729 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260501 13:58:21.054140 3734 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 15dd70d7c5494c3c911f0a780dc9a2af)
I20260501 13:58:21.054584 3734 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:21.056097 3734 leader_election.cc:290] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255)
I20260501 13:58:21.056923 3163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:21.057251 3237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:21.058604 3294 leader_election.cc:304] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0f8911dd5d431d9d8d3124a4456262; no voters: 0a1d2cd221924677a3ef81b081e5e016, 15dd70d7c5494c3c911f0a780dc9a2af
W20260501 13:58:21.058763 3729 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
I20260501 13:58:21.059196 3734 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260501 13:58:21.059991 3729 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.16.131
I20260501 13:58:21.100143 3729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.16.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.16.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.16.131
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:58:21.101624 3729 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:58:21.103597 3729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260501 13:58:21.110670 3735 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 15dd70d7c5494c3c911f0a780dc9a2af)
I20260501 13:58:21.111045 3735 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:21.112610 3735 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:21.113422 3237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:21.113726 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:21.115667 3139 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016; no voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:21.116427 3735 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260501 13:58:21.118716 3737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:58:21.229835 3720 heartbeater.cc:499] Master 127.1.16.189:44255 was elected leader, sending a full tablet report...
I20260501 13:58:22.478919 3746 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 15dd70d7c5494c3c911f0a780dc9a2af)
I20260501 13:58:22.479702 3746 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:22.482698 3237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:22.486639 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:22.487746 3318 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a1d2cd221924677a3ef81b081e5e016 in term 2.
I20260501 13:58:22.489223 3139 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262; no voters: 15dd70d7c5494c3c911f0a780dc9a2af
I20260501 13:58:22.491870 3746 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:22.492018 3747 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260501 13:58:22.493635 3747 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 15dd70d7c5494c3c911f0a780dc9a2af)
I20260501 13:58:22.494017 3747 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 2 FOLLOWER]: Advancing to term 3
I20260501 13:58:22.498847 3745 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 15dd70d7c5494c3c911f0a780dc9a2af)
I20260501 13:58:22.499768 3745 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:22.503976 3163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:22.506721 3237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:22.508587 3745 leader_election.cc:290] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255)
I20260501 13:58:22.515671 3747 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:22.517117 3163 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b0f8911dd5d431d9d8d3124a4456262 in current term 3: Already voted for candidate 0a1d2cd221924677a3ef81b081e5e016 in this term.
I20260501 13:58:22.520120 3237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af"
I20260501 13:58:22.522677 3293 leader_election.cc:304] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0f8911dd5d431d9d8d3124a4456262; no voters: 0a1d2cd221924677a3ef81b081e5e016, 15dd70d7c5494c3c911f0a780dc9a2af
I20260501 13:58:22.523303 3747 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 3 election: Requested vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:22.523254 3318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262"
I20260501 13:58:22.524186 3745 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 2 FOLLOWER]: Advancing to term 3
I20260501 13:58:22.524977 3139 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016; no voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:22.526207 3747 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260501 13:58:22.534817 3745 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260501 13:58:22.560602 3163 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 3 FOLLOWER]: Rejecting Update request from peer 15dd70d7c5494c3c911f0a780dc9a2af for earlier term 2. Current term is 3. Ops: []
I20260501 13:58:22.562933 3748 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260501 13:58:22.564237 3748 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 2 LEADER]: Stepping down as leader of term 2
I20260501 13:58:22.564628 3748 raft_consensus.cc:740] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 15dd70d7c5494c3c911f0a780dc9a2af, State: Running, Role: LEADER
I20260501 13:58:22.565462 3748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:22.568423 3748 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 2 FOLLOWER]: Advancing to term 3
W20260501 13:58:22.519780 3736 debug-util.cc:398] Leaking SignalData structure 0x7b0800028ca0 after lost signal to thread 3729
W20260501 13:58:22.650533 3729 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.532s user 0.616s sys 0.859s
W20260501 13:58:21.119040 3738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:22.651553 3729 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.533s user 0.617s sys 0.859s
W20260501 13:58:22.652262 3739 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1531 milliseconds
I20260501 13:58:22.652815 3729 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260501 13:58:22.652872 3740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:58:22.656553 3729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:58:22.659157 3729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:58:22.660575 3729 hybrid_clock.cc:648] HybridClock initialized: now 1777643902660535 us; error 63 us; skew 500 ppm
I20260501 13:58:22.661360 3729 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:58:22.668437 3729 webserver.cc:492] Webserver started at http://127.1.16.131:41777/ using document root <none> and password file <none>
I20260501 13:58:22.669911 3729 fs_manager.cc:362] Metadata directory not provided
I20260501 13:58:22.670208 3729 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:58:22.670801 3729 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 13:58:22.677122 3729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/data/instance:
uuid: "5c239ccbdc554619a237d78e958ffbc2"
format_stamp: "Formatted at 2026-05-01 13:58:22 on dist-test-slave-nqxd"
I20260501 13:58:22.678781 3729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/instance:
uuid: "5c239ccbdc554619a237d78e958ffbc2"
format_stamp: "Formatted at 2026-05-01 13:58:22 on dist-test-slave-nqxd"
I20260501 13:58:22.686980 3729 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260501 13:58:22.693059 3754 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:22.694200 3729 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260501 13:58:22.694552 3729 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal
uuid: "5c239ccbdc554619a237d78e958ffbc2"
format_stamp: "Formatted at 2026-05-01 13:58:22 on dist-test-slave-nqxd"
I20260501 13:58:22.694904 3729 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:58:22.762416 3729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:58:22.763983 3729 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:58:22.764427 3729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:58:22.767000 3729 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 13:58:22.771145 3729 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 13:58:22.771363 3729 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:22.771605 3729 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 13:58:22.771761 3729 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:22.917871 3729 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.131:38685
I20260501 13:58:22.918001 3866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.131:38685 every 8 connection(s)
I20260501 13:58:22.921491 3729 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/data/info.pb
I20260501 13:58:22.928421 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 3729
I20260501 13:58:22.928959 1090 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/instance
I20260501 13:58:22.969127 3868 heartbeater.cc:344] Connected to a master server at 127.1.16.189:44255
I20260501 13:58:22.969684 3868 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:22.970417 3869 heartbeater.cc:344] Connected to a master server at 127.1.16.190:38561
I20260501 13:58:22.970765 3869 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:22.971127 3868 heartbeater.cc:507] Master 127.1.16.189:44255 requested a full tablet report, sending...
I20260501 13:58:22.971844 3869 heartbeater.cc:507] Master 127.1.16.190:38561 requested a full tablet report, sending...
I20260501 13:58:22.974364 3154 ts_manager.cc:194] Registered new tserver with Master: 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:22.974514 3227 ts_manager.cc:194] Registered new tserver with Master: 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:22.974561 3867 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38253
I20260501 13:58:22.974891 3867 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:22.975811 3867 heartbeater.cc:507] Master 127.1.16.188:38253 requested a full tablet report, sending...
I20260501 13:58:22.980810 3308 ts_manager.cc:194] Registered new tserver with Master: 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:22.995692 1090 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 13:58:23.085769 3884 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:23.086287 3884 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:23.089811 3884 leader_election.cc:290] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255)
I20260501 13:58:23.090369 3163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:23.090883 3163 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b0f8911dd5d431d9d8d3124a4456262 in term 3.
I20260501 13:58:23.091114 3237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:23.091625 3237 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b0f8911dd5d431d9d8d3124a4456262 in term 3.
I20260501 13:58:23.091778 3293 leader_election.cc:304] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262; no voters:
I20260501 13:58:23.092404 3884 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260501 13:58:23.092662 3884 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:23.092895 3884 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 3 FOLLOWER]: Advancing to term 4
I20260501 13:58:23.097718 3884 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:23.099185 3884 leader_election.cc:290] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 4 election: Requested vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255)
I20260501 13:58:23.099910 3163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016"
I20260501 13:58:23.100102 3237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af"
I20260501 13:58:23.100402 3163 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 3 FOLLOWER]: Advancing to term 4
I20260501 13:58:23.100646 3237 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 3 FOLLOWER]: Advancing to term 4
I20260501 13:58:23.105731 3237 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b0f8911dd5d431d9d8d3124a4456262 in term 4.
I20260501 13:58:23.106814 3294 leader_election.cc:304] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262; no voters:
I20260501 13:58:23.107592 3884 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 4 FOLLOWER]: Leader election won for term 4
I20260501 13:58:23.108624 3884 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 4 LEADER]: Becoming Leader. State: Replica: 6b0f8911dd5d431d9d8d3124a4456262, State: Running, Role: LEADER
I20260501 13:58:23.109400 3884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:23.113140 3885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0f8911dd5d431d9d8d3124a4456262. Latest consensus state: current_term: 4 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:23.113976 3885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: LEADER
I20260501 13:58:23.115492 3887 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 13:58:23.120855 3887 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 13:58:23.123456 3887 catalog_manager.cc:1269] Loaded cluster ID: 4281e294e62c41c781e2041e1b24de7c
I20260501 13:58:23.123669 3887 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 13:58:23.126749 3887 catalog_manager.cc:1514] Loading token signing keys...
I20260501 13:58:23.129384 3887 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Loaded TSK: 0
I20260501 13:58:23.130524 3887 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 13:58:23.133059 3887 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260501 13:58:23.135097 3163 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b0f8911dd5d431d9d8d3124a4456262 in term 4.
I20260501 13:58:23.135921 3887 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260501 13:58:23.162663 3308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41662:
name: "default.Table_a9e0f183ed664f379d44ea82dab95fa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:23.164157 3307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41662:
name: "default.Table_27ad4801c45e4f2da414165d40bf4ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:23.172145 3306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41662:
name: "default.Table_4c6e426a58b5461bbfaef6abc2ffd23f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:23.173949 3305 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41662:
name: "default.Table_5c8ebfa533ee4e9a94d8200053bfe4c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:58:23.176399 3308 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a9e0f183ed664f379d44ea82dab95fa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:23.183717 3307 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_27ad4801c45e4f2da414165d40bf4ae4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:23.183842 3306 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4c6e426a58b5461bbfaef6abc2ffd23f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:23.194906 3305 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:23.208488 1090 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu with pid 3264
W20260501 13:58:23.234917 3878 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.16.188:38253)
W20260501 13:58:23.247462 3719 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
W20260501 13:58:23.500561 3867 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
W20260501 13:58:23.501292 3561 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:23.779091 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.188:38253
--webserver_interface=127.1.16.188
--webserver_port=46681
--builtin_ntp_servers=127.1.16.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--hive_metastore_uris=thrift://127.0.0.1:45913
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260501 13:58:23.783823 3911 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:23.784288 3911 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:23.787115 3163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:23.787735 3163 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15dd70d7c5494c3c911f0a780dc9a2af in term 4.
I20260501 13:58:23.789213 3213 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016, 15dd70d7c5494c3c911f0a780dc9a2af; no voters:
I20260501 13:58:23.790796 3912 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260501 13:58:23.791159 3912 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:23.791432 3912 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 4 FOLLOWER]: Advancing to term 5
I20260501 13:58:23.797257 3911 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:23.800900 3912 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:23.803648 3912 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 5 election: Requested vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:23.806284 3163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016"
I20260501 13:58:23.806831 3163 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 4 FOLLOWER]: Advancing to term 5
I20260501 13:58:23.813457 3163 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15dd70d7c5494c3c911f0a780dc9a2af in term 5.
W20260501 13:58:23.805800 3213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260501 13:58:23.818903 3213 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016, 15dd70d7c5494c3c911f0a780dc9a2af; no voters:
I20260501 13:58:23.819796 3912 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 5 FOLLOWER]: Leader election won for term 5
I20260501 13:58:23.820233 3912 raft_consensus.cc:697] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 5 LEADER]: Becoming Leader. State: Replica: 15dd70d7c5494c3c911f0a780dc9a2af, State: Running, Role: LEADER
I20260501 13:58:23.820974 3912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:23.830773 3911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15dd70d7c5494c3c911f0a780dc9a2af. Latest consensus state: current_term: 5 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:23.831522 3911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: LEADER
I20260501 13:58:23.833866 3917 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
W20260501 13:58:23.837879 3213 leader_election.cc:336] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253): Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
W20260501 13:58:23.838886 3213 leader_election.cc:336] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253): Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111)
I20260501 13:58:23.842417 3917 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 13:58:23.846151 3917 catalog_manager.cc:1269] Loaded cluster ID: 4281e294e62c41c781e2041e1b24de7c
I20260501 13:58:23.846427 3917 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 13:58:23.850798 3917 catalog_manager.cc:1514] Loading token signing keys...
I20260501 13:58:23.854489 3917 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Loaded TSK: 0
I20260501 13:58:23.855487 3917 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 13:58:23.858947 3917 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260501 13:58:23.862519 3917 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260501 13:58:23.890530 3228 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_5c8ebfa533ee4e9a94d8200053bfe4c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:23.893579 3227 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_4c6e426a58b5461bbfaef6abc2ffd23f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:23.896093 3226 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_27ad4801c45e4f2da414165d40bf4ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:23.898761 3225 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_a9e0f183ed664f379d44ea82dab95fa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:58:23.931298 3213 consensus_peers.cc:597] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af -> Peer 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253): Couldn't send request to peer 6b0f8911dd5d431d9d8d3124a4456262. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260501 13:58:23.933382 3163 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 5 FOLLOWER]: Refusing update from remote peer 15dd70d7c5494c3c911f0a780dc9a2af: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260501 13:58:23.939146 3911 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260501 13:58:23.960772 3919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15dd70d7c5494c3c911f0a780dc9a2af. Latest consensus state: current_term: 5 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:23.961472 3919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:23.963547 3918 mvcc.cc:204] Tried to move back new op lower bound from 7281229430475198464 to 7281229430057959424. Current Snapshot: MvccSnapshot[applied={T|T < 7281229430475198464}]
I20260501 13:58:23.972425 3919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:23.972860 3911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:23.973311 3919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:23.973542 3911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: LEADER
W20260501 13:58:23.977135 3228 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:23.984766 3227 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4c6e426a58b5461bbfaef6abc2ffd23f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:23.987562 3226 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_27ad4801c45e4f2da414165d40bf4ae4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:23.989610 3225 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a9e0f183ed664f379d44ea82dab95fa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:23.996409 3224 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.1.16.131:34833
I20260501 13:58:24.125471 3226 catalog_manager.cc:2174] created HMS catalog entry for table [id=c8b517f0258c4d35a40759e478d314ee]
I20260501 13:58:24.148083 3226 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50472:
table { table_name: "default.Table_27ad4801c45e4f2da414165d40bf4ae4" } new_table_name: "default.Table_e3c7989801224024a8debb2eda1686e8" modify_external_catalogs: true
I20260501 13:58:24.220538 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 4ea3394b02fb4d7cac308ed81c93f1ce (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.222447 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea3394b02fb4d7cac308ed81c93f1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:24.221416 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 4ea3394b02fb4d7cac308ed81c93f1ce (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.224356 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea3394b02fb4d7cac308ed81c93f1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:24.230885 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 4ea3394b02fb4d7cac308ed81c93f1ce (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.232863 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea3394b02fb4d7cac308ed81c93f1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:24.300822 3937 tablet_bootstrap.cc:492] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:24.310179 3936 tablet_bootstrap.cc:492] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:24.310945 3935 tablet_bootstrap.cc:492] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:24.320760 3935 tablet_bootstrap.cc:654] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:24.323746 3224 master_service.cc:954] ReplaceTablet: received request to replace tablet 4ea3394b02fb4d7cac308ed81c93f1ce from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:24.325461 3937 tablet_bootstrap.cc:654] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:24.327795 3937 log.cc:826] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: Log is configured to *not* fsync() on all Append() calls
I20260501 13:58:24.329893 3935 log.cc:826] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122: Log is configured to *not* fsync() on all Append() calls
I20260501 13:58:24.330011 3223 master_service.cc:954] ReplaceTablet: received request to replace tablet 4ea3394b02fb4d7cac308ed81c93f1ce from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:24.332980 3936 tablet_bootstrap.cc:654] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:24.341599 3937 tablet_bootstrap.cc:492] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:24.342163 3937 ts_tablet_manager.cc:1403] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.006s
I20260501 13:58:24.348659 3222 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_1d615bc9ac7a4db39b4d4afb4ef38d8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:24.350630 3935 tablet_bootstrap.cc:492] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:24.351207 3935 ts_tablet_manager.cc:1403] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.007s
I20260501 13:58:24.365365 3224 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 4ea3394b02fb4d7cac308ed81c93f1ce
I20260501 13:58:24.372206 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 4ea3394b02fb4d7cac308ed81c93f1ce with delete_type TABLET_DATA_DELETED (replaced by tablet bcc1ed167cb74157ba83df1faf62eac6) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:24.372936 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 4ea3394b02fb4d7cac308ed81c93f1ce with delete_type TABLET_DATA_DELETED (replaced by tablet bcc1ed167cb74157ba83df1faf62eac6) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:24.374724 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 4ea3394b02fb4d7cac308ed81c93f1ce with delete_type TABLET_DATA_DELETED (replaced by tablet bcc1ed167cb74157ba83df1faf62eac6) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:24.378643 3224 catalog_manager.cc:6741] ReplaceTablet: tablet 4ea3394b02fb4d7cac308ed81c93f1ce deleted and replaced by tablet bcc1ed167cb74157ba83df1faf62eac6
W20260501 13:58:24.381629 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 4ea3394b02fb4d7cac308ed81c93f1ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea3394b02fb4d7cac308ed81c93f1ce already in progress: creating tablet
W20260501 13:58:24.391773 3914 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:58:24.392565 3914 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 13:58:24.393250 3914 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 13:58:24.393105 3213 catalog_manager.cc:4982] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet 4ea3394b02fb4d7cac308ed81c93f1ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea3394b02fb4d7cac308ed81c93f1ce already in progress: creating tablet
I20260501 13:58:24.395856 3936 log.cc:826] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1: Log is configured to *not* fsync() on all Append() calls
I20260501 13:58:24.400345 3495 tablet_service.cc:1511] Processing CreateTablet for tablet bcc1ed167cb74157ba83df1faf62eac6 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.407653 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcc1ed167cb74157ba83df1faf62eac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:24.417662 3802 tablet_service.cc:1511] Processing CreateTablet for tablet bcc1ed167cb74157ba83df1faf62eac6 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.418862 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcc1ed167cb74157ba83df1faf62eac6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:58:24.424578 3213 catalog_manager.cc:4982] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet 4ea3394b02fb4d7cac308ed81c93f1ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea3394b02fb4d7cac308ed81c93f1ce already in progress: creating tablet
I20260501 13:58:24.425350 3935 raft_consensus.cc:359] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.426221 3935 raft_consensus.cc:385] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:24.416914 3654 tablet_service.cc:1511] Processing CreateTablet for tablet bcc1ed167cb74157ba83df1faf62eac6 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.433001 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcc1ed167cb74157ba83df1faf62eac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:24.433199 3937 raft_consensus.cc:359] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.434152 3937 raft_consensus.cc:385] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:24.434437 3937 raft_consensus.cc:740] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:24.435575 3937 consensus_queue.cc:260] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.437434 3935 raft_consensus.cc:740] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:24.438551 3935 consensus_queue.cc:260] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.449191 3937 ts_tablet_manager.cc:1434] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.107s user 0.034s sys 0.004s
I20260501 13:58:24.450742 3562 heartbeater.cc:499] Master 127.1.16.189:44255 was elected leader, sending a full tablet report...
I20260501 13:58:24.452381 3868 heartbeater.cc:499] Master 127.1.16.189:44255 was elected leader, sending a full tablet report...
W20260501 13:58:24.458863 3914 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260501 13:58:24.459362 3914 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260501 13:58:24.459797 3914 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 13:58:24.460201 3914 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:58:24.460618 3914 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 13:58:24.461000 3914 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260501 13:58:24.463451 3872 tablet.cc:2404] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:58:24.463636 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 4ea3394b02fb4d7cac308ed81c93f1ce with delete_type TABLET_DATA_DELETED (replaced by tablet bcc1ed167cb74157ba83df1faf62eac6) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:24.465149 3935 ts_tablet_manager.cc:1434] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.114s user 0.029s sys 0.008s
W20260501 13:58:24.465862 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 4ea3394b02fb4d7cac308ed81c93f1ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea3394b02fb4d7cac308ed81c93f1ce already in progress: creating tablet
I20260501 13:58:24.473495 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 4ea3394b02fb4d7cac308ed81c93f1ce with delete_type TABLET_DATA_DELETED (replaced by tablet bcc1ed167cb74157ba83df1faf62eac6) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:24.474450 3943 tablet_replica.cc:333] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:24.476728 3943 raft_consensus.cc:2243] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:24.477324 3943 raft_consensus.cc:2272] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:24.480329 3943 ts_tablet_manager.cc:1916] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:24.482982 3936 tablet_bootstrap.cc:492] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:24.483525 3936 ts_tablet_manager.cc:1403] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.174s user 0.015s sys 0.004s
I20260501 13:58:24.490286 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 4ea3394b02fb4d7cac308ed81c93f1ce with delete_type TABLET_DATA_DELETED (replaced by tablet bcc1ed167cb74157ba83df1faf62eac6) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:24.491461 3941 tablet_replica.cc:333] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:24.492273 3941 raft_consensus.cc:2243] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:24.492866 3941 raft_consensus.cc:2272] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:24.496302 3941 ts_tablet_manager.cc:1916] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:24.513222 3943 ts_tablet_manager.cc:1929] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:24.515511 3937 tablet_bootstrap.cc:492] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:24.516693 3935 tablet_bootstrap.cc:492] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:24.521297 3941 ts_tablet_manager.cc:1929] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:24.521756 3941 log.cc:1199] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/4ea3394b02fb4d7cac308ed81c93f1ce
I20260501 13:58:24.522069 3937 tablet_bootstrap.cc:654] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:24.522663 3941 ts_tablet_manager.cc:1950] T 4ea3394b02fb4d7cac308ed81c93f1ce P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:24.526005 3943 log.cc:1199] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/4ea3394b02fb4d7cac308ed81c93f1ce
I20260501 13:58:24.527086 3943 ts_tablet_manager.cc:1950] T 4ea3394b02fb4d7cac308ed81c93f1ce P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:24.530556 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 4ea3394b02fb4d7cac308ed81c93f1ce (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:24.525283 3935 tablet_bootstrap.cc:654] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:24.535089 3937 tablet_bootstrap.cc:492] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:24.535200 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 4ea3394b02fb4d7cac308ed81c93f1ce (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:24.535616 3937 ts_tablet_manager.cc:1403] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.007s
I20260501 13:58:24.538300 3937 raft_consensus.cc:359] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.538993 3937 raft_consensus.cc:385] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:24.539281 3937 raft_consensus.cc:740] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:24.540060 3936 raft_consensus.cc:359] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.540889 3936 raft_consensus.cc:385] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:24.541177 3936 raft_consensus.cc:740] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:24.542008 3936 consensus_queue.cc:260] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.544342 3937 consensus_queue.cc:260] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.548873 3720 heartbeater.cc:499] Master 127.1.16.189:44255 was elected leader, sending a full tablet report...
I20260501 13:58:24.549989 3936 ts_tablet_manager.cc:1434] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.066s user 0.026s sys 0.008s
I20260501 13:58:24.550940 3936 tablet_bootstrap.cc:492] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:24.552989 3937 ts_tablet_manager.cc:1434] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.017s user 0.000s sys 0.006s
I20260501 13:58:24.556866 3936 tablet_bootstrap.cc:654] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:24.562232 3914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45913
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.188:38253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.16.188
--webserver_port=46681
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:58:24.564488 3914 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:58:24.566893 3914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260501 13:58:24.568063 3936 tablet_bootstrap.cc:492] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:24.568614 3936 ts_tablet_manager.cc:1403] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.005s
I20260501 13:58:24.571496 3936 raft_consensus.cc:359] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.572446 3936 raft_consensus.cc:385] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:24.572786 3936 raft_consensus.cc:740] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:24.573498 3936 consensus_queue.cc:260] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.576124 3936 ts_tablet_manager.cc:1434] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260501 13:58:24.579857 3224 master_service.cc:954] ReplaceTablet: received request to replace tablet bcc1ed167cb74157ba83df1faf62eac6 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:24.591892 3223 master_service.cc:954] ReplaceTablet: received request to replace tablet bcc1ed167cb74157ba83df1faf62eac6 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:24.602483 3224 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet bcc1ed167cb74157ba83df1faf62eac6
W20260501 13:58:24.603400 3953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:58:24.603905 3224 catalog_manager.cc:6741] ReplaceTablet: tablet bcc1ed167cb74157ba83df1faf62eac6 deleted and replaced by tablet 282f15ef3a644ff785393e6eee7a429b
I20260501 13:58:24.604274 3935 tablet_bootstrap.cc:492] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:24.604717 3935 ts_tablet_manager.cc:1403] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.010s
I20260501 13:58:24.606894 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet bcc1ed167cb74157ba83df1faf62eac6 with delete_type TABLET_DATA_DELETED (replaced by tablet 282f15ef3a644ff785393e6eee7a429b) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:24.606210 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet bcc1ed167cb74157ba83df1faf62eac6 with delete_type TABLET_DATA_DELETED (replaced by tablet 282f15ef3a644ff785393e6eee7a429b) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:24.607615 3943 tablet_replica.cc:333] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:24.607887 3942 tablet_replica.cc:333] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:24.608255 3943 raft_consensus.cc:2243] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:24.608707 3943 raft_consensus.cc:2272] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:24.608753 3942 raft_consensus.cc:2243] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:24.609305 3942 raft_consensus.cc:2272] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:24.611204 3943 ts_tablet_manager.cc:1916] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:24.613973 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 4ea3394b02fb4d7cac308ed81c93f1ce with delete_type TABLET_DATA_DELETED (replaced by tablet bcc1ed167cb74157ba83df1faf62eac6) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:24.620811 3935 raft_consensus.cc:359] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.621613 3935 raft_consensus.cc:385] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:24.621912 3935 raft_consensus.cc:740] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:24.622756 3935 consensus_queue.cc:260] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.625375 3935 ts_tablet_manager.cc:1434] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.020s user 0.004s sys 0.004s
I20260501 13:58:24.630663 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet bcc1ed167cb74157ba83df1faf62eac6 with delete_type TABLET_DATA_DELETED (replaced by tablet 282f15ef3a644ff785393e6eee7a429b) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:24.631145 3942 ts_tablet_manager.cc:1916] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:24.631484 3941 tablet_replica.cc:333] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:24.632133 3941 raft_consensus.cc:2243] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:24.632587 3941 raft_consensus.cc:2272] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:24.644037 3943 ts_tablet_manager.cc:1929] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:24.644443 3943 log.cc:1199] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/bcc1ed167cb74157ba83df1faf62eac6
I20260501 13:58:24.645093 3943 ts_tablet_manager.cc:1950] T bcc1ed167cb74157ba83df1faf62eac6 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:24.647709 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet bcc1ed167cb74157ba83df1faf62eac6 (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:24.646612 3941 ts_tablet_manager.cc:1916] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:24.672101 3942 ts_tablet_manager.cc:1929] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:24.672065 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 282f15ef3a644ff785393e6eee7a429b (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.672544 3942 log.cc:1199] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/bcc1ed167cb74157ba83df1faf62eac6
I20260501 13:58:24.673210 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 282f15ef3a644ff785393e6eee7a429b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:24.673456 3942 ts_tablet_manager.cc:1950] T bcc1ed167cb74157ba83df1faf62eac6 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:24.674682 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 282f15ef3a644ff785393e6eee7a429b (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.675741 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 282f15ef3a644ff785393e6eee7a429b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:24.679345 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 282f15ef3a644ff785393e6eee7a429b (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.680346 3942 tablet_replica.cc:333] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:24.680542 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 282f15ef3a644ff785393e6eee7a429b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:24.681013 3942 raft_consensus.cc:2243] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:24.681511 3942 raft_consensus.cc:2272] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:24.691041 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet bcc1ed167cb74157ba83df1faf62eac6 (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:24.696249 3942 ts_tablet_manager.cc:1916] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
13:58:24.702 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_a9e0f183ed664f379d44ea82dab95fa2 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at jdk.proxy2/jdk.proxy2.$Proxy27.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
W20260501 13:58:24.706312 3225 catalog_manager.cc:2170] Already present: failed to create HMS catalog entry for table [id=d84b514cc8e245cda25bbdf5a4d45642]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_a9e0f183ed664f379d44ea82dab95fa2 already exists)
I20260501 13:58:24.708616 3935 tablet_bootstrap.cc:492] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:24.716096 3935 tablet_bootstrap.cc:654] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:24.720144 3941 ts_tablet_manager.cc:1929] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:24.721900 3225 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_b07edfe247554299af96b76fee9a5b4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:24.726500 3941 log.cc:1199] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/bcc1ed167cb74157ba83df1faf62eac6
I20260501 13:58:24.727221 3937 tablet_bootstrap.cc:492] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:24.727341 3941 ts_tablet_manager.cc:1950] T bcc1ed167cb74157ba83df1faf62eac6 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:24.729398 3936 tablet_bootstrap.cc:492] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:24.730093 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet bcc1ed167cb74157ba83df1faf62eac6 (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:24.733310 3937 tablet_bootstrap.cc:654] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:24.735649 3936 tablet_bootstrap.cc:654] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:24.741444 3935 tablet_bootstrap.cc:492] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:24.741918 3935 ts_tablet_manager.cc:1403] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.000s
I20260501 13:58:24.744617 3935 raft_consensus.cc:359] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.745307 3935 raft_consensus.cc:385] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:24.745568 3935 raft_consensus.cc:740] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:24.750239 3937 tablet_bootstrap.cc:492] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:24.750697 3937 ts_tablet_manager.cc:1403] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.009s
I20260501 13:58:24.753383 3937 raft_consensus.cc:359] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.754042 3937 raft_consensus.cc:385] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:24.754320 3937 raft_consensus.cc:740] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:24.759111 3937 consensus_queue.cc:260] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.760735 3936 tablet_bootstrap.cc:492] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:24.761178 3936 ts_tablet_manager.cc:1403] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.007s
I20260501 13:58:24.763830 3936 raft_consensus.cc:359] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.764484 3936 raft_consensus.cc:385] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:24.764767 3936 raft_consensus.cc:740] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:24.765427 3936 consensus_queue.cc:260] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.746199 3935 consensus_queue.cc:260] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:24.774655 3935 ts_tablet_manager.cc:1434] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260501 13:58:24.778030 3936 ts_tablet_manager.cc:1434] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260501 13:58:24.779651 3937 ts_tablet_manager.cc:1434] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20260501 13:58:24.781191 3942 ts_tablet_manager.cc:1929] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:24.781623 3942 log.cc:1199] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/4ea3394b02fb4d7cac308ed81c93f1ce
I20260501 13:58:24.782245 3942 ts_tablet_manager.cc:1950] T 4ea3394b02fb4d7cac308ed81c93f1ce P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:24.792062 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 4ea3394b02fb4d7cac308ed81c93f1ce (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:24.839716 3223 master_service.cc:954] ReplaceTablet: received request to replace tablet 282f15ef3a644ff785393e6eee7a429b from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:24.851437 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet 282f15ef3a644ff785393e6eee7a429b from {username='slave'} at 127.0.0.1:50472
W20260501 13:58:24.861773 3568 tablet.cc:2404] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:58:24.866933 3223 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 282f15ef3a644ff785393e6eee7a429b
I20260501 13:58:24.868666 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 282f15ef3a644ff785393e6eee7a429b with delete_type TABLET_DATA_DELETED (replaced by tablet 1518e4e6c34349a9894c8b46b2d5f020) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:24.869441 3941 tablet_replica.cc:333] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:24.870113 3941 raft_consensus.cc:2243] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:24.870719 3941 raft_consensus.cc:2272] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:24.872861 3941 ts_tablet_manager.cc:1916] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:24.875247 3227 catalog_manager.cc:2174] created HMS catalog entry for table [id=0f9b011bb04e4620a47cd4853e6c4129]
I20260501 13:58:24.877198 3223 catalog_manager.cc:6741] ReplaceTablet: tablet 282f15ef3a644ff785393e6eee7a429b deleted and replaced by tablet 1518e4e6c34349a9894c8b46b2d5f020
I20260501 13:58:24.891821 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 282f15ef3a644ff785393e6eee7a429b with delete_type TABLET_DATA_DELETED (replaced by tablet 1518e4e6c34349a9894c8b46b2d5f020) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:24.891799 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 282f15ef3a644ff785393e6eee7a429b with delete_type TABLET_DATA_DELETED (replaced by tablet 1518e4e6c34349a9894c8b46b2d5f020) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:24.892524 3942 tablet_replica.cc:333] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:24.893131 3942 raft_consensus.cc:2243] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:24.893642 3942 raft_consensus.cc:2272] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:24.894433 3943 tablet_replica.cc:333] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:24.895397 3943 raft_consensus.cc:2243] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:24.895784 3943 raft_consensus.cc:2272] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:24.897475 3942 ts_tablet_manager.cc:1916] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:24.899945 3943 ts_tablet_manager.cc:1916] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:24.940915 3942 ts_tablet_manager.cc:1929] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:24.948762 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 1518e4e6c34349a9894c8b46b2d5f020 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.950091 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1518e4e6c34349a9894c8b46b2d5f020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:24.957158 3943 ts_tablet_manager.cc:1929] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:24.957451 3227 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50472 (request call id 36) took 1064 ms. Trace:
I20260501 13:58:24.957840 3227 rpcz_store.cc:276] 0501 13:58:23.893050 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:23.893190 (+ 140us) service_pool.cc:224] Handling call
0501 13:58:23.985302 (+ 92112us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:24.008498 (+ 23196us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:24.875203 (+866705us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:24.956744 (+ 81541us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:24.957067 (+ 323us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:24.957370 (+ 303us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:24.877505 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:24.878422 (+ 917us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:24.878449 (+ 27us) write_op.cc:435] Acquired schema lock
0501 13:58:24.878480 (+ 31us) tablet.cc:662] Decoding operations
0501 13:58:24.878775 (+ 295us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:24.878842 (+ 67us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:24.878874 (+ 32us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:24.879096 (+ 222us) tablet.cc:701] Row locks acquired
0501 13:58:24.879113 (+ 17us) write_op.cc:260] PREPARE: finished
0501 13:58:24.879242 (+ 129us) write_op.cc:270] Start()
0501 13:58:24.879347 (+ 105us) write_op.cc:276] Timestamp: P: 1777643904879194 usec, L: 0
0501 13:58:24.879368 (+ 21us) op_driver.cc:348] REPLICATION: starting
0501 13:58:24.879995 (+ 627us) log.cc:844] Serialized 432 byte log entry
0501 13:58:24.935799 (+ 55804us) op_driver.cc:464] REPLICATION: finished
0501 13:58:24.937843 (+ 2044us) write_op.cc:301] APPLY: starting
0501 13:58:24.937895 (+ 52us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:24.938002 (+ 107us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:24.938079 (+ 77us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:24.938419 (+ 340us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:24.955523 (+ 17104us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:24.955574 (+ 51us) write_op.cc:312] APPLY: finished
0501 13:58:24.955779 (+ 205us) log.cc:844] Serialized 28 byte log entry
0501 13:58:24.955943 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:24.956026 (+ 83us) write_op.cc:454] Released schema lock
0501 13:58:24.956057 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":696562,"Hive Metastore.run_cpu_time_us":688,"Hive Metastore.run_wall_time_us":168952,"child_traces":[["op",{"apply.queue_time_us":1909,"num_ops":2,"prepare.queue_time_us":687,"prepare.run_cpu_time_us":2271,"prepare.run_wall_time_us":4816,"raft.queue_time_us":950,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":18,"replication_time_us":56350,"threads_started":1}]]}
I20260501 13:58:24.972425 3941 ts_tablet_manager.cc:1929] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:24.972847 3941 log.cc:1199] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/282f15ef3a644ff785393e6eee7a429b
I20260501 13:58:24.973500 3941 ts_tablet_manager.cc:1950] T 282f15ef3a644ff785393e6eee7a429b P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:24.967590 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 1518e4e6c34349a9894c8b46b2d5f020 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.975358 3942 log.cc:1199] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/282f15ef3a644ff785393e6eee7a429b
I20260501 13:58:24.975940 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1518e4e6c34349a9894c8b46b2d5f020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:24.976017 3942 ts_tablet_manager.cc:1950] T 282f15ef3a644ff785393e6eee7a429b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:24.976755 3943 log.cc:1199] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/282f15ef3a644ff785393e6eee7a429b
I20260501 13:58:24.977432 3943 ts_tablet_manager.cc:1950] T 282f15ef3a644ff785393e6eee7a429b P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:24.969596 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 1518e4e6c34349a9894c8b46b2d5f020 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:24.979363 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1518e4e6c34349a9894c8b46b2d5f020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:24.984835 3223 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50472:
table { table_name: "default.Table_4c6e426a58b5461bbfaef6abc2ffd23f" } new_table_name: "default.Table_2840ee60b88a494f9b40b448d78b0000" modify_external_catalogs: true
I20260501 13:58:24.986107 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 282f15ef3a644ff785393e6eee7a429b (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:24.987031 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 282f15ef3a644ff785393e6eee7a429b (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:24.987679 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 282f15ef3a644ff785393e6eee7a429b (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:25.036926 3228 catalog_manager.cc:2174] created HMS catalog entry for table [id=096f180e34c94777bb6cf87fded9c6c3]
I20260501 13:58:25.070103 3936 tablet_bootstrap.cc:492] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:25.077095 3936 tablet_bootstrap.cc:654] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.077818 3228 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50472 (request call id 35) took 1188 ms. Trace:
I20260501 13:58:25.078171 3228 rpcz_store.cc:276] 0501 13:58:23.889761 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:23.889907 (+ 146us) service_pool.cc:224] Handling call
0501 13:58:23.977778 (+ 87871us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:24.009458 (+ 31680us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:25.036879 (+1027421us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:25.077074 (+ 40195us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:25.077452 (+ 378us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:25.077754 (+ 302us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:25.039979 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:25.040519 (+ 540us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:25.040551 (+ 32us) write_op.cc:435] Acquired schema lock
0501 13:58:25.040564 (+ 13us) tablet.cc:662] Decoding operations
0501 13:58:25.040909 (+ 345us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:25.040953 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:25.040980 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:25.041199 (+ 219us) tablet.cc:701] Row locks acquired
0501 13:58:25.041217 (+ 18us) write_op.cc:260] PREPARE: finished
0501 13:58:25.041370 (+ 153us) write_op.cc:270] Start()
0501 13:58:25.041476 (+ 106us) write_op.cc:276] Timestamp: P: 1777643905041320 usec, L: 0
0501 13:58:25.041495 (+ 19us) op_driver.cc:348] REPLICATION: starting
0501 13:58:25.042053 (+ 558us) log.cc:844] Serialized 432 byte log entry
0501 13:58:25.072528 (+ 30475us) op_driver.cc:464] REPLICATION: finished
0501 13:58:25.074617 (+ 2089us) write_op.cc:301] APPLY: starting
0501 13:58:25.074672 (+ 55us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:25.074774 (+ 102us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:25.074852 (+ 78us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:25.075214 (+ 362us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:25.075352 (+ 138us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:25.075400 (+ 48us) write_op.cc:312] APPLY: finished
0501 13:58:25.075575 (+ 175us) log.cc:844] Serialized 28 byte log entry
0501 13:58:25.075698 (+ 123us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:25.075761 (+ 63us) write_op.cc:454] Released schema lock
0501 13:58:25.075790 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":864691,"Hive Metastore.run_cpu_time_us":720,"Hive Metastore.run_wall_time_us":161853,"child_traces":[["op",{"apply.queue_time_us":1954,"num_ops":2,"prepare.queue_time_us":1214,"prepare.run_cpu_time_us":2434,"prepare.run_wall_time_us":2434,"raft.queue_time_us":1056,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":16,"replication_time_us":30949}]]}
I20260501 13:58:25.088392 3494 tablet_service.cc:1511] Processing CreateTablet for tablet f9a995604b954b4daf041614a12a606c (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.089555 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9a995604b954b4daf041614a12a606c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.093914 3936 tablet_bootstrap.cc:492] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:25.094396 3936 ts_tablet_manager.cc:1403] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.002s
I20260501 13:58:25.094717 3937 tablet_bootstrap.cc:492] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:25.096400 3935 tablet_bootstrap.cc:492] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:25.097862 3228 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50472:
table { table_name: "default.Table_5c8ebfa533ee4e9a94d8200053bfe4c0" } modify_external_catalogs: true
I20260501 13:58:25.100423 3654 tablet_service.cc:1511] Processing CreateTablet for tablet f9a995604b954b4daf041614a12a606c (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.100993 3937 tablet_bootstrap.cc:654] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.101841 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9a995604b954b4daf041614a12a606c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.102454 3935 tablet_bootstrap.cc:654] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.108112 3936 raft_consensus.cc:359] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.108855 3936 raft_consensus.cc:385] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.109160 3936 raft_consensus.cc:740] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.110280 3936 consensus_queue.cc:260] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.111948 3802 tablet_service.cc:1511] Processing CreateTablet for tablet f9a995604b954b4daf041614a12a606c (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.112231 3228 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50472:
table { table_name: "default.Table_5c8ebfa533ee4e9a94d8200053bfe4c0" } modify_external_catalogs: true
I20260501 13:58:25.113101 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9a995604b954b4daf041614a12a606c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.115600 3936 ts_tablet_manager.cc:1434] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260501 13:58:25.135010 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 1518e4e6c34349a9894c8b46b2d5f020 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:25.138078 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet 1518e4e6c34349a9894c8b46b2d5f020 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:25.138234 3937 tablet_bootstrap.cc:492] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:25.138674 3937 ts_tablet_manager.cc:1403] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.044s user 0.015s sys 0.000s
I20260501 13:58:25.138769 3935 tablet_bootstrap.cc:492] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:25.139189 3935 ts_tablet_manager.cc:1403] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.043s user 0.014s sys 0.002s
I20260501 13:58:25.141487 3937 raft_consensus.cc:359] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.142166 3937 raft_consensus.cc:385] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.143074 3935 raft_consensus.cc:359] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.143733 3935 raft_consensus.cc:385] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.144025 3935 raft_consensus.cc:740] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.144786 3935 consensus_queue.cc:260] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.147225 3935 ts_tablet_manager.cc:1434] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260501 13:58:25.142448 3937 raft_consensus.cc:740] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.156068 3937 consensus_queue.cc:260] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.163749 3935 tablet_bootstrap.cc:492] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:25.168071 3219 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_1f445634a0fc4676a2e7bf208b0ea1de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:25.169960 3937 ts_tablet_manager.cc:1434] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20260501 13:58:25.172714 3935 tablet_bootstrap.cc:654] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.186744 3936 tablet_bootstrap.cc:492] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:25.193060 3936 tablet_bootstrap.cc:654] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:25.198944 3722 tablet.cc:2404] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:58:25.202113 3935 tablet_bootstrap.cc:492] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:25.202697 3935 ts_tablet_manager.cc:1403] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.002s
I20260501 13:58:25.205591 3935 raft_consensus.cc:359] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.206408 3935 raft_consensus.cc:385] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.206768 3935 raft_consensus.cc:740] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.207461 3935 consensus_queue.cc:260] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.210417 3950 raft_consensus.cc:493] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:25.211103 3950 raft_consensus.cc:515] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.216581 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 1518e4e6c34349a9894c8b46b2d5f020 with delete_type TABLET_DATA_DELETED (replaced by tablet fe149add97b347dc94a9856c68aefdd2) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:25.217128 3950 leader_election.cc:290] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:25.217324 3943 tablet_replica.cc:333] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:25.218027 3943 raft_consensus.cc:2243] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:25.218523 3943 raft_consensus.cc:2272] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:25.219755 3935 ts_tablet_manager.cc:1434] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260501 13:58:25.224054 3943 ts_tablet_manager.cc:1916] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:25.212177 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 1518e4e6c34349a9894c8b46b2d5f020 with delete_type TABLET_DATA_DELETED (replaced by tablet fe149add97b347dc94a9856c68aefdd2) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:25.226012 3941 tablet_replica.cc:333] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:25.226747 3941 raft_consensus.cc:2243] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:25.227239 3941 raft_consensus.cc:2272] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:25.228026 3936 tablet_bootstrap.cc:492] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:25.228585 3936 ts_tablet_manager.cc:1403] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.000s
I20260501 13:58:25.229713 3941 ts_tablet_manager.cc:1916] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:25.231499 3936 raft_consensus.cc:359] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.234651 3936 raft_consensus.cc:385] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.218204 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 1518e4e6c34349a9894c8b46b2d5f020
I20260501 13:58:25.236734 3936 raft_consensus.cc:740] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.237494 3936 consensus_queue.cc:260] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.238684 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 1518e4e6c34349a9894c8b46b2d5f020 with delete_type TABLET_DATA_DELETED (replaced by tablet fe149add97b347dc94a9856c68aefdd2) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:25.244138 3936 ts_tablet_manager.cc:1434] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260501 13:58:25.244520 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 1518e4e6c34349a9894c8b46b2d5f020 with delete_type TABLET_DATA_DELETED (replaced by tablet fe149add97b347dc94a9856c68aefdd2) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:25.246524 3227 catalog_manager.cc:6741] ReplaceTablet: tablet 1518e4e6c34349a9894c8b46b2d5f020 deleted and replaced by tablet fe149add97b347dc94a9856c68aefdd2
I20260501 13:58:25.267087 3937 tablet_bootstrap.cc:492] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:25.274060 3937 tablet_bootstrap.cc:654] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.289930 3937 tablet_bootstrap.cc:492] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:25.290210 3220 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_ed51787806aa40069d50f787cc85ae8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:25.292028 3937 ts_tablet_manager.cc:1403] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.000s
I20260501 13:58:25.295708 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1518e4e6c34349a9894c8b46b2d5f020" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
W20260501 13:58:25.297922 3609 leader_election.cc:343] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: Tablet not RUNNING: STOPPED
I20260501 13:58:25.299446 3494 tablet_service.cc:1511] Processing CreateTablet for tablet 998090e07eaf41e69eade304e7cb3946 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.301033 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998090e07eaf41e69eade304e7cb3946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.305459 3495 tablet_service.cc:1511] Processing CreateTablet for tablet fe149add97b347dc94a9856c68aefdd2 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.306702 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe149add97b347dc94a9856c68aefdd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.315271 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 998090e07eaf41e69eade304e7cb3946 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.316354 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998090e07eaf41e69eade304e7cb3946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.319010 3937 raft_consensus.cc:359] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.322007 3937 raft_consensus.cc:385] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.322854 3937 raft_consensus.cc:740] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.328013 3937 consensus_queue.cc:260] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.346726 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 1518e4e6c34349a9894c8b46b2d5f020 with delete_type TABLET_DATA_DELETED (replaced by tablet fe149add97b347dc94a9856c68aefdd2) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:25.347595 3942 tablet_replica.cc:333] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:25.347862 3801 tablet_service.cc:1511] Processing CreateTablet for tablet fe149add97b347dc94a9856c68aefdd2 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.348656 3942 raft_consensus.cc:2243] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:25.349061 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe149add97b347dc94a9856c68aefdd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.349143 3942 raft_consensus.cc:2272] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:25.350862 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 998090e07eaf41e69eade304e7cb3946 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.351361 3942 ts_tablet_manager.cc:1916] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:25.351962 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998090e07eaf41e69eade304e7cb3946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.341010 3937 ts_tablet_manager.cc:1434] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20260501 13:58:25.365245 3653 tablet_service.cc:1511] Processing CreateTablet for tablet fe149add97b347dc94a9856c68aefdd2 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.366572 3653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe149add97b347dc94a9856c68aefdd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.382761 3937 tablet_bootstrap.cc:492] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:25.385519 3942 ts_tablet_manager.cc:1929] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:25.385936 3942 log.cc:1199] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/1518e4e6c34349a9894c8b46b2d5f020
I20260501 13:58:25.386669 3942 ts_tablet_manager.cc:1950] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:25.394727 3943 ts_tablet_manager.cc:1929] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:25.395136 3943 log.cc:1199] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/1518e4e6c34349a9894c8b46b2d5f020
I20260501 13:58:25.396548 3943 ts_tablet_manager.cc:1950] T 1518e4e6c34349a9894c8b46b2d5f020 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:25.399092 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 1518e4e6c34349a9894c8b46b2d5f020 (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
W20260501 13:58:25.403630 3213 catalog_manager.cc:4969] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet 1518e4e6c34349a9894c8b46b2d5f020 because the tablet was not found. No further retry: Not found: Tablet not found: 1518e4e6c34349a9894c8b46b2d5f020
I20260501 13:58:25.404932 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 1518e4e6c34349a9894c8b46b2d5f020 (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
W20260501 13:58:25.412233 3608 leader_election.cc:343] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 1518e4e6c34349a9894c8b46b2d5f020
I20260501 13:58:25.412789 3608 leader_election.cc:304] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122
W20260501 13:58:25.413635 3608 raft_consensus.cc:2686] T 1518e4e6c34349a9894c8b46b2d5f020 P 6af0cedc7f3c47299b93e6e92d8e02d1: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260501 13:58:25.415356 3937 tablet_bootstrap.cc:654] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.419924 3936 tablet_bootstrap.cc:492] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:25.428275 3936 tablet_bootstrap.cc:654] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.434386 3935 tablet_bootstrap.cc:492] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:25.434670 3224 master_service.cc:954] ReplaceTablet: received request to replace tablet f9a995604b954b4daf041614a12a606c from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:25.442982 3935 tablet_bootstrap.cc:654] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.451797 3941 ts_tablet_manager.cc:1929] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:25.452435 3941 log.cc:1199] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/1518e4e6c34349a9894c8b46b2d5f020
I20260501 13:58:25.453250 3941 ts_tablet_manager.cc:1950] T 1518e4e6c34349a9894c8b46b2d5f020 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:25.454375 3947 raft_consensus.cc:493] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:25.455188 3947 raft_consensus.cc:515] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.463828 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet fe149add97b347dc94a9856c68aefdd2 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:25.466096 3947 leader_election.cc:290] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:25.470726 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 1518e4e6c34349a9894c8b46b2d5f020 (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
W20260501 13:58:25.474620 3214 catalog_manager.cc:4969] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 1518e4e6c34349a9894c8b46b2d5f020 because the tablet was not found. No further retry: Not found: Tablet not found: 1518e4e6c34349a9894c8b46b2d5f020
I20260501 13:58:25.478797 3936 tablet_bootstrap.cc:492] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:25.479274 3936 ts_tablet_manager.cc:1403] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.004s
I20260501 13:58:25.481997 3936 raft_consensus.cc:359] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.498833 3936 raft_consensus.cc:385] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.499167 3936 raft_consensus.cc:740] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
W20260501 13:58:25.501322 3213 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.188:38253, user_credentials={real_user=slave}} blocked reactor thread for 48144.9us
I20260501 13:58:25.502233 3936 consensus_queue.cc:260] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.507761 3936 ts_tablet_manager.cc:1434] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260501 13:58:25.508714 3936 tablet_bootstrap.cc:492] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:25.510049 3935 tablet_bootstrap.cc:492] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:25.510203 3224 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet f9a995604b954b4daf041614a12a606c
I20260501 13:58:25.510912 3937 tablet_bootstrap.cc:492] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:25.511384 3937 ts_tablet_manager.cc:1403] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.129s user 0.048s sys 0.019s
I20260501 13:58:25.511762 3224 catalog_manager.cc:6741] ReplaceTablet: tablet f9a995604b954b4daf041614a12a606c deleted and replaced by tablet cf9af59b508b41c0a7bd4efd9bbc369b
I20260501 13:58:25.517620 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet f9a995604b954b4daf041614a12a606c with delete_type TABLET_DATA_DELETED (replaced by tablet cf9af59b508b41c0a7bd4efd9bbc369b) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:25.517858 3937 raft_consensus.cc:359] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.518520 3941 tablet_replica.cc:333] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:25.519219 3937 raft_consensus.cc:385] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.519217 3941 raft_consensus.cc:2243] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:25.519615 3937 raft_consensus.cc:740] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.519690 3941 raft_consensus.cc:2272] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:25.522158 3941 ts_tablet_manager.cc:1916] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:25.524156 3221 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet fe149add97b347dc94a9856c68aefdd2
I20260501 13:58:25.524495 3936 tablet_bootstrap.cc:654] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.525642 3221 catalog_manager.cc:6741] ReplaceTablet: tablet fe149add97b347dc94a9856c68aefdd2 deleted and replaced by tablet d341bb06f3b24efa84f6b84751cb4459
I20260501 13:58:25.525408 3937 consensus_queue.cc:260] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.529177 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet f9a995604b954b4daf041614a12a606c with delete_type TABLET_DATA_DELETED (replaced by tablet cf9af59b508b41c0a7bd4efd9bbc369b) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:25.530308 3942 tablet_replica.cc:333] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:25.531080 3942 raft_consensus.cc:2243] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:25.531567 3942 raft_consensus.cc:2272] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:25.530313 3937 ts_tablet_manager.cc:1434] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20260501 13:58:25.518630 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet f9a995604b954b4daf041614a12a606c with delete_type TABLET_DATA_DELETED (replaced by tablet cf9af59b508b41c0a7bd4efd9bbc369b) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:25.536019 3943 tablet_replica.cc:333] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:25.536134 3937 tablet_bootstrap.cc:492] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:25.536770 3943 raft_consensus.cc:2243] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:25.537263 3943 raft_consensus.cc:2272] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:25.539794 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet fe149add97b347dc94a9856c68aefdd2 with delete_type TABLET_DATA_DELETED (replaced by tablet d341bb06f3b24efa84f6b84751cb4459) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:25.542562 3937 tablet_bootstrap.cc:654] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.544633 3943 ts_tablet_manager.cc:1916] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:25.550879 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet fe149add97b347dc94a9856c68aefdd2 with delete_type TABLET_DATA_DELETED (replaced by tablet d341bb06f3b24efa84f6b84751cb4459) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:25.551095 3942 ts_tablet_manager.cc:1916] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:25.558555 3937 tablet_bootstrap.cc:492] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:25.559067 3937 ts_tablet_manager.cc:1403] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.007s
I20260501 13:58:25.561563 3937 raft_consensus.cc:359] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.562381 3937 raft_consensus.cc:385] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.562704 3937 raft_consensus.cc:740] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.563333 3937 consensus_queue.cc:260] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.563855 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9a995604b954b4daf041614a12a606c" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
W20260501 13:58:25.565527 3448 leader_election.cc:343] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Illegal state: Tablet not RUNNING: STOPPED
I20260501 13:58:25.571118 3950 raft_consensus.cc:493] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:25.571347 3937 ts_tablet_manager.cc:1434] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260501 13:58:25.571615 3950 raft_consensus.cc:515] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.573474 3950 leader_election.cc:290] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:25.574486 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe149add97b347dc94a9856c68aefdd2" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
W20260501 13:58:25.576066 3609 leader_election.cc:343] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:25.581878 3494 tablet_service.cc:1511] Processing CreateTablet for tablet d341bb06f3b24efa84f6b84751cb4459 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.582551 3801 tablet_service.cc:1511] Processing CreateTablet for tablet cf9af59b508b41c0a7bd4efd9bbc369b (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.583300 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d341bb06f3b24efa84f6b84751cb4459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.583632 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf9af59b508b41c0a7bd4efd9bbc369b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.581794 3495 tablet_service.cc:1511] Processing CreateTablet for tablet cf9af59b508b41c0a7bd4efd9bbc369b (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.589749 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf9af59b508b41c0a7bd4efd9bbc369b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.604351 3802 tablet_service.cc:1511] Processing CreateTablet for tablet d341bb06f3b24efa84f6b84751cb4459 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.605523 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d341bb06f3b24efa84f6b84751cb4459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.611685 3943 ts_tablet_manager.cc:1929] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:25.612227 3943 log.cc:1199] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/f9a995604b954b4daf041614a12a606c
I20260501 13:58:25.617136 3943 ts_tablet_manager.cc:1950] T f9a995604b954b4daf041614a12a606c P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:25.617751 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9a995604b954b4daf041614a12a606c" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
W20260501 13:58:25.619643 3448 leader_election.cc:343] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: Tablet not RUNNING: STOPPED
I20260501 13:58:25.619843 3943 tablet_replica.cc:333] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:25.620055 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet f9a995604b954b4daf041614a12a606c (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:25.620177 3448 leader_election.cc:304] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:25.620846 3943 raft_consensus.cc:2243] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260501 13:58:25.621057 3448 raft_consensus.cc:2686] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260501 13:58:25.621443 3943 raft_consensus.cc:2272] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:25.621783 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe149add97b347dc94a9856c68aefdd2" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:25.623836 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet fe149add97b347dc94a9856c68aefdd2 with delete_type TABLET_DATA_DELETED (replaced by tablet d341bb06f3b24efa84f6b84751cb4459) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:25.624366 3943 ts_tablet_manager.cc:1916] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:25.624691 3935 ts_tablet_manager.cc:1403] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.191s user 0.002s sys 0.013s
I20260501 13:58:25.631268 3654 tablet_service.cc:1511] Processing CreateTablet for tablet cf9af59b508b41c0a7bd4efd9bbc369b (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.632433 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf9af59b508b41c0a7bd4efd9bbc369b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.652122 3943 ts_tablet_manager.cc:1929] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:25.652653 3943 log.cc:1199] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/fe149add97b347dc94a9856c68aefdd2
I20260501 13:58:25.653879 3937 tablet_bootstrap.cc:492] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:25.655431 3653 tablet_service.cc:1511] Processing CreateTablet for tablet d341bb06f3b24efa84f6b84751cb4459 (DEFAULT_TABLE table=default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.656652 3653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d341bb06f3b24efa84f6b84751cb4459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.663069 3937 tablet_bootstrap.cc:654] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.653421 3943 ts_tablet_manager.cc:1950] T fe149add97b347dc94a9856c68aefdd2 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
W20260501 13:58:25.665673 3608 leader_election.cc:343] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:25.666076 3608 leader_election.cc:304] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:25.668112 3935 raft_consensus.cc:359] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.668810 3935 raft_consensus.cc:385] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.669106 3935 raft_consensus.cc:740] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.672180 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet fe149add97b347dc94a9856c68aefdd2 (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:25.679785 3937 tablet_bootstrap.cc:492] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:25.680328 3937 ts_tablet_manager.cc:1403] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.002s
I20260501 13:58:25.683072 3950 raft_consensus.cc:2749] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:25.686291 3936 tablet_bootstrap.cc:492] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:25.686758 3936 ts_tablet_manager.cc:1403] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.178s user 0.037s sys 0.025s
I20260501 13:58:25.686422 3937 raft_consensus.cc:359] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.687740 3937 raft_consensus.cc:385] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.688170 3937 raft_consensus.cc:740] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.689167 3937 consensus_queue.cc:260] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.689429 3936 raft_consensus.cc:359] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.691416 3936 raft_consensus.cc:385] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.691669 3936 raft_consensus.cc:740] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.703353 3935 consensus_queue.cc:260] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.705507 3937 ts_tablet_manager.cc:1434] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260501 13:58:25.708779 3936 consensus_queue.cc:260] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.712738 3936 ts_tablet_manager.cc:1434] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.026s user 0.005s sys 0.001s
I20260501 13:58:25.713795 3936 tablet_bootstrap.cc:492] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:25.716046 3935 ts_tablet_manager.cc:1434] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.091s user 0.005s sys 0.001s
I20260501 13:58:25.717233 3935 tablet_bootstrap.cc:492] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:25.718508 3950 raft_consensus.cc:493] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:25.718952 3950 raft_consensus.cc:515] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.720785 3950 leader_election.cc:290] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:25.721284 3942 ts_tablet_manager.cc:1929] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:25.721756 3942 log.cc:1199] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/f9a995604b954b4daf041614a12a606c
I20260501 13:58:25.723469 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "998090e07eaf41e69eade304e7cb3946" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:25.724144 3515 raft_consensus.cc:2468] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:25.730016 3937 tablet_bootstrap.cc:492] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:25.738147 3936 tablet_bootstrap.cc:654] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.738922 3942 ts_tablet_manager.cc:1950] T f9a995604b954b4daf041614a12a606c P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:25.739670 3937 tablet_bootstrap.cc:654] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.741731 3942 tablet_replica.cc:333] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:25.741813 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 998090e07eaf41e69eade304e7cb3946 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:25.755072 3609 leader_election.cc:304] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:25.760144 3935 tablet_bootstrap.cc:654] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.762879 3942 raft_consensus.cc:2243] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:25.763552 3942 raft_consensus.cc:2272] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:25.770581 3941 ts_tablet_manager.cc:1929] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:25.771131 3941 log.cc:1199] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/f9a995604b954b4daf041614a12a606c
I20260501 13:58:25.776399 3942 ts_tablet_manager.cc:1916] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:25.782315 3950 raft_consensus.cc:2804] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:25.782764 3950 raft_consensus.cc:493] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:25.783053 3941 ts_tablet_manager.cc:1950] T f9a995604b954b4daf041614a12a606c P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:25.783249 3950 raft_consensus.cc:3060] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:25.789819 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet f9a995604b954b4daf041614a12a606c (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:25.791712 3224 master_service.cc:954] ReplaceTablet: received request to replace tablet 998090e07eaf41e69eade304e7cb3946 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:25.792291 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 998090e07eaf41e69eade304e7cb3946 with delete_type TABLET_DATA_DELETED (replaced by tablet e8ede8efc21a4f2ca821810be62c5e07) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:25.801669 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 998090e07eaf41e69eade304e7cb3946
I20260501 13:58:25.803382 3227 catalog_manager.cc:6741] ReplaceTablet: tablet 998090e07eaf41e69eade304e7cb3946 deleted and replaced by tablet e8ede8efc21a4f2ca821810be62c5e07
I20260501 13:58:25.805562 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 998090e07eaf41e69eade304e7cb3946 with delete_type TABLET_DATA_DELETED (replaced by tablet e8ede8efc21a4f2ca821810be62c5e07) from {username='slave'} at 127.0.0.1:58038
W20260501 13:58:25.809365 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet fe149add97b347dc94a9856c68aefdd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe149add97b347dc94a9856c68aefdd2 already in progress: creating tablet
I20260501 13:58:25.811380 3941 tablet_replica.cc:333] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:25.811345 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 998090e07eaf41e69eade304e7cb3946 with delete_type TABLET_DATA_DELETED (replaced by tablet e8ede8efc21a4f2ca821810be62c5e07) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:25.812119 3943 tablet_replica.cc:333] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:25.812143 3941 raft_consensus.cc:2243] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:25.812633 3941 raft_consensus.cc:2272] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:25.812808 3943 raft_consensus.cc:2243] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:25.813268 3943 raft_consensus.cc:2272] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:25.813764 3935 tablet_bootstrap.cc:492] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:25.814213 3935 ts_tablet_manager.cc:1403] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.097s user 0.013s sys 0.000s
I20260501 13:58:25.815814 3943 ts_tablet_manager.cc:1916] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:25.816810 3935 raft_consensus.cc:359] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.817534 3935 raft_consensus.cc:385] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.817802 3935 raft_consensus.cc:740] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.834327 3943 ts_tablet_manager.cc:1929] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:25.837868 3943 log.cc:1199] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/998090e07eaf41e69eade304e7cb3946
I20260501 13:58:25.839540 3943 ts_tablet_manager.cc:1950] T 998090e07eaf41e69eade304e7cb3946 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:25.830420 3941 ts_tablet_manager.cc:1916] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:25.842144 3950 raft_consensus.cc:515] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.842741 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 998090e07eaf41e69eade304e7cb3946 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:25.844588 3935 consensus_queue.cc:260] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.845041 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "998090e07eaf41e69eade304e7cb3946" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
W20260501 13:58:25.846685 3609 leader_election.cc:343] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: Tablet not RUNNING: STOPPED
I20260501 13:58:25.848510 3495 tablet_service.cc:1511] Processing CreateTablet for tablet e8ede8efc21a4f2ca821810be62c5e07 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.849668 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ede8efc21a4f2ca821810be62c5e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.860790 3935 ts_tablet_manager.cc:1434] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20260501 13:58:25.867161 3935 tablet_bootstrap.cc:492] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:25.870692 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet fe149add97b347dc94a9856c68aefdd2 with delete_type TABLET_DATA_DELETED (replaced by tablet d341bb06f3b24efa84f6b84751cb4459) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:25.900460 3942 ts_tablet_manager.cc:1929] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:25.907546 3935 tablet_bootstrap.cc:654] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:25.908671 3942 log.cc:1199] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/fe149add97b347dc94a9856c68aefdd2
I20260501 13:58:25.910542 3802 tablet_service.cc:1511] Processing CreateTablet for tablet e8ede8efc21a4f2ca821810be62c5e07 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.911850 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ede8efc21a4f2ca821810be62c5e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.910962 3942 ts_tablet_manager.cc:1950] T fe149add97b347dc94a9856c68aefdd2 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:25.927522 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet f9a995604b954b4daf041614a12a606c (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:25.929481 3653 tablet_service.cc:1558] Processing DeleteTablet for tablet 998090e07eaf41e69eade304e7cb3946 with delete_type TABLET_DATA_DELETED (replaced by tablet e8ede8efc21a4f2ca821810be62c5e07) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:25.931555 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet fe149add97b347dc94a9856c68aefdd2 (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:25.931391 3652 tablet_service.cc:1511] Processing CreateTablet for tablet e8ede8efc21a4f2ca821810be62c5e07 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:25.933071 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ede8efc21a4f2ca821810be62c5e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:25.934010 3942 tablet_replica.cc:333] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:25.936684 3942 raft_consensus.cc:2243] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:25.935307 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 998090e07eaf41e69eade304e7cb3946 with delete_type TABLET_DATA_DELETED (replaced by tablet e8ede8efc21a4f2ca821810be62c5e07) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:25.934801 3950 leader_election.cc:290] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:25.943151 3941 ts_tablet_manager.cc:1929] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:25.943861 3942 raft_consensus.cc:2272] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:58:25.950515 3608 leader_election.cc:343] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 998090e07eaf41e69eade304e7cb3946
I20260501 13:58:25.951905 3936 tablet_bootstrap.cc:492] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:25.948592 3935 tablet_bootstrap.cc:492] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:25.953234 3935 ts_tablet_manager.cc:1403] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.086s user 0.011s sys 0.000s
I20260501 13:58:25.955651 3935 raft_consensus.cc:359] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.956905 3935 raft_consensus.cc:385] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.957525 3936 ts_tablet_manager.cc:1403] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.244s user 0.020s sys 0.014s
I20260501 13:58:25.958016 3935 raft_consensus.cc:740] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.960767 3936 raft_consensus.cc:359] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.961848 3935 consensus_queue.cc:260] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.956465 3941 log.cc:1199] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/998090e07eaf41e69eade304e7cb3946
I20260501 13:58:25.967212 3941 ts_tablet_manager.cc:1950] T 998090e07eaf41e69eade304e7cb3946 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:25.970130 3935 ts_tablet_manager.cc:1434] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20260501 13:58:25.973539 3935 tablet_bootstrap.cc:492] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
W20260501 13:58:25.956615 3608 leader_election.cc:343] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 998090e07eaf41e69eade304e7cb3946
I20260501 13:58:25.974375 3608 leader_election.cc:304] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122
W20260501 13:58:25.975199 3608 raft_consensus.cc:2686] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260501 13:58:25.978089 3936 raft_consensus.cc:385] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.978605 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 998090e07eaf41e69eade304e7cb3946 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
W20260501 13:58:25.979379 3214 catalog_manager.cc:4969] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 998090e07eaf41e69eade304e7cb3946 because the tablet was not found. No further retry: Not found: Tablet not found: 998090e07eaf41e69eade304e7cb3946
I20260501 13:58:25.986620 3937 tablet_bootstrap.cc:492] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:25.987078 3937 ts_tablet_manager.cc:1403] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.257s user 0.053s sys 0.077s
I20260501 13:58:25.978741 3936 raft_consensus.cc:740] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.989499 3937 raft_consensus.cc:359] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.990139 3937 raft_consensus.cc:385] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:25.990406 3937 raft_consensus.cc:740] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:25.991524 3937 consensus_queue.cc:260] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.968405 3942 ts_tablet_manager.cc:1916] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:25.993475 3936 consensus_queue.cc:260] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:25.977627 3941 tablet_replica.cc:333] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:25.997205 3941 raft_consensus.cc:2243] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:26.001124 3936 ts_tablet_manager.cc:1434] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.042s user 0.003s sys 0.001s
I20260501 13:58:26.002168 3941 raft_consensus.cc:2272] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.004948 3937 ts_tablet_manager.cc:1434] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260501 13:58:26.009225 3945 raft_consensus.cc:493] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:26.009830 3945 raft_consensus.cc:515] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.014225 3941 ts_tablet_manager.cc:1916] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.018100 3936 tablet_bootstrap.cc:492] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:26.020088 3224 master_service.cc:954] ReplaceTablet: received request to replace tablet cf9af59b508b41c0a7bd4efd9bbc369b from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:26.033388 3945 leader_election.cc:290] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:26.026079 3935 tablet_bootstrap.cc:654] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.039280 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet fe149add97b347dc94a9856c68aefdd2 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:26.044991 3936 tablet_bootstrap.cc:654] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.050035 3224 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet cf9af59b508b41c0a7bd4efd9bbc369b
I20260501 13:58:26.050529 3937 tablet_bootstrap.cc:492] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:26.052435 3224 catalog_manager.cc:6741] ReplaceTablet: tablet cf9af59b508b41c0a7bd4efd9bbc369b deleted and replaced by tablet f2ebffc1b9e849319e05351f926208bb
I20260501 13:58:26.055936 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet cf9af59b508b41c0a7bd4efd9bbc369b with delete_type TABLET_DATA_DELETED (replaced by tablet f2ebffc1b9e849319e05351f926208bb) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:26.057627 3943 tablet_replica.cc:333] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:26.058238 3943 raft_consensus.cc:2243] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:26.058588 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet cf9af59b508b41c0a7bd4efd9bbc369b with delete_type TABLET_DATA_DELETED (replaced by tablet f2ebffc1b9e849319e05351f926208bb) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:26.058696 3943 raft_consensus.cc:2272] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.058859 3935 tablet_bootstrap.cc:492] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:26.062907 3935 ts_tablet_manager.cc:1403] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.003s
I20260501 13:58:26.063817 3943 ts_tablet_manager.cc:1916] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.081692 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet cf9af59b508b41c0a7bd4efd9bbc369b with delete_type TABLET_DATA_DELETED (replaced by tablet f2ebffc1b9e849319e05351f926208bb) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:26.093238 3936 tablet_bootstrap.cc:492] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:26.095604 3937 tablet_bootstrap.cc:654] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.104274 3936 ts_tablet_manager.cc:1403] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.096s user 0.010s sys 0.001s
I20260501 13:58:26.086068 3935 raft_consensus.cc:359] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.109633 3935 raft_consensus.cc:385] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.113495 3935 raft_consensus.cc:740] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.122946 3935 consensus_queue.cc:260] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.128567 3935 ts_tablet_manager.cc:1434] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.045s user 0.003s sys 0.004s
I20260501 13:58:26.109392 3943 ts_tablet_manager.cc:1929] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.129760 3943 log.cc:1199] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/cf9af59b508b41c0a7bd4efd9bbc369b
I20260501 13:58:26.137444 3943 ts_tablet_manager.cc:1950] T cf9af59b508b41c0a7bd4efd9bbc369b P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:26.140460 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet cf9af59b508b41c0a7bd4efd9bbc369b (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:26.141449 3937 tablet_bootstrap.cc:492] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:26.141860 3937 ts_tablet_manager.cc:1403] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.000s
I20260501 13:58:26.142788 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d341bb06f3b24efa84f6b84751cb4459" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:26.143646 3674 raft_consensus.cc:2468] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:26.144255 3937 raft_consensus.cc:359] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.144865 3937 raft_consensus.cc:385] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.145121 3937 raft_consensus.cc:740] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.145757 3756 leader_election.cc:304] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:26.146724 3945 raft_consensus.cc:2804] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:26.147094 3945 raft_consensus.cc:493] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:26.147413 3945 raft_consensus.cc:3060] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:26.152163 3652 tablet_service.cc:1511] Processing CreateTablet for tablet f2ebffc1b9e849319e05351f926208bb (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.121695 3936 raft_consensus.cc:359] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.153169 3936 raft_consensus.cc:385] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.131433 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d341bb06f3b24efa84f6b84751cb4459" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:26.154047 3515 raft_consensus.cc:2468] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:26.154372 3945 raft_consensus.cc:515] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.147306 3495 tablet_service.cc:1511] Processing CreateTablet for tablet f2ebffc1b9e849319e05351f926208bb (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.155920 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ebffc1b9e849319e05351f926208bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.159739 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d341bb06f3b24efa84f6b84751cb4459" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:26.160384 3515 raft_consensus.cc:3060] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:26.161329 3945 leader_election.cc:290] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:26.145684 3937 consensus_queue.cc:260] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.162789 3802 tablet_service.cc:1511] Processing CreateTablet for tablet f2ebffc1b9e849319e05351f926208bb (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.164239 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ebffc1b9e849319e05351f926208bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.176309 3515 raft_consensus.cc:2468] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:26.177838 3757 leader_election.cc:304] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:26.179235 3945 raft_consensus.cc:2804] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:26.183017 3945 raft_consensus.cc:697] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:26.163040 3937 ts_tablet_manager.cc:1434] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20260501 13:58:26.184417 3945 consensus_queue.cc:237] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.195360 3935 tablet_bootstrap.cc:492] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:26.200232 3941 ts_tablet_manager.cc:1929] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.153321 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ebffc1b9e849319e05351f926208bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.160800 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d341bb06f3b24efa84f6b84751cb4459" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:26.206809 3674 raft_consensus.cc:3060] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:26.219048 3674 raft_consensus.cc:2468] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:26.222607 3936 raft_consensus.cc:740] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.179282 3942 ts_tablet_manager.cc:1929] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.223402 3942 log.cc:1199] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/998090e07eaf41e69eade304e7cb3946
I20260501 13:58:26.224249 3942 ts_tablet_manager.cc:1950] T 998090e07eaf41e69eade304e7cb3946 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:26.226897 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 998090e07eaf41e69eade304e7cb3946 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:26.223814 3936 consensus_queue.cc:260] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.231887 3937 tablet_bootstrap.cc:492] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:26.234859 3936 ts_tablet_manager.cc:1434] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.125s user 0.001s sys 0.006s
I20260501 13:58:26.237684 3937 tablet_bootstrap.cc:654] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.238678 3941 log.cc:1199] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/fe149add97b347dc94a9856c68aefdd2
I20260501 13:58:26.240903 3941 ts_tablet_manager.cc:1950] T fe149add97b347dc94a9856c68aefdd2 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:26.249305 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet f2ebffc1b9e849319e05351f926208bb from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:26.253656 3224 catalog_manager.cc:5671] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } }
W20260501 13:58:26.257018 3213 catalog_manager.cc:4969] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet 998090e07eaf41e69eade304e7cb3946 because the tablet was not found. No further retry: Not found: Tablet not found: 998090e07eaf41e69eade304e7cb3946
I20260501 13:58:26.258886 3936 tablet_bootstrap.cc:492] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:26.264106 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet f2ebffc1b9e849319e05351f926208bb from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:26.272624 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet fe149add97b347dc94a9856c68aefdd2 (table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:26.280129 3221 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet f2ebffc1b9e849319e05351f926208bb
I20260501 13:58:26.282085 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet f2ebffc1b9e849319e05351f926208bb with delete_type TABLET_DATA_DELETED (replaced by tablet 49ec9f6f25ea477796d52d45b19c5317) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:26.285696 3221 catalog_manager.cc:6741] ReplaceTablet: tablet f2ebffc1b9e849319e05351f926208bb deleted and replaced by tablet 49ec9f6f25ea477796d52d45b19c5317
I20260501 13:58:26.288673 3941 tablet_replica.cc:333] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:26.289182 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet f2ebffc1b9e849319e05351f926208bb with delete_type TABLET_DATA_DELETED (replaced by tablet 49ec9f6f25ea477796d52d45b19c5317) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:26.291741 3941 raft_consensus.cc:2243] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260501 13:58:26.292176 3213 catalog_manager.cc:4982] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet f2ebffc1b9e849319e05351f926208bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2ebffc1b9e849319e05351f926208bb already in progress: creating tablet
I20260501 13:58:26.292387 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet f2ebffc1b9e849319e05351f926208bb with delete_type TABLET_DATA_DELETED (replaced by tablet 49ec9f6f25ea477796d52d45b19c5317) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:26.298161 3937 tablet_bootstrap.cc:492] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:26.298635 3937 ts_tablet_manager.cc:1403] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.005s
I20260501 13:58:26.290884 3935 tablet_bootstrap.cc:654] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.301343 3942 tablet_replica.cc:333] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:26.301023 3937 raft_consensus.cc:359] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.301632 3937 raft_consensus.cc:385] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.301867 3937 raft_consensus.cc:740] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.302016 3942 raft_consensus.cc:2243] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:26.302445 3942 raft_consensus.cc:2272] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.302855 3937 consensus_queue.cc:260] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.304546 3937 ts_tablet_manager.cc:1434] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260501 13:58:26.312368 3942 ts_tablet_manager.cc:1916] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.319772 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet f2ebffc1b9e849319e05351f926208bb with delete_type TABLET_DATA_DELETED (replaced by tablet 49ec9f6f25ea477796d52d45b19c5317) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:26.320400 3941 raft_consensus.cc:2272] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.320533 3943 tablet_replica.cc:333] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:26.321137 3943 raft_consensus.cc:2243] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:26.321569 3943 raft_consensus.cc:2272] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.323448 3943 ts_tablet_manager.cc:1916] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.330740 3936 tablet_bootstrap.cc:654] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.349651 3941 ts_tablet_manager.cc:1916] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.362352 3943 ts_tablet_manager.cc:1929] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.362890 3935 tablet_bootstrap.cc:492] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:26.362926 3943 log.cc:1199] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/f2ebffc1b9e849319e05351f926208bb
I20260501 13:58:26.363327 3935 ts_tablet_manager.cc:1403] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.168s user 0.010s sys 0.001s
I20260501 13:58:26.363696 3943 ts_tablet_manager.cc:1950] T f2ebffc1b9e849319e05351f926208bb P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
W20260501 13:58:26.364555 3213 consensus_peers.cc:597] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af -> Peer 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253): Couldn't send request to peer 6b0f8911dd5d431d9d8d3124a4456262. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.188:38253: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260501 13:58:26.365599 3935 raft_consensus.cc:359] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.366212 3935 raft_consensus.cc:385] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.366480 3935 raft_consensus.cc:740] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.367133 3935 consensus_queue.cc:260] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.368957 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 49ec9f6f25ea477796d52d45b19c5317 (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.369962 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49ec9f6f25ea477796d52d45b19c5317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.371876 3936 tablet_bootstrap.cc:492] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:26.379555 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 49ec9f6f25ea477796d52d45b19c5317 (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.379695 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet f2ebffc1b9e849319e05351f926208bb (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:26.380565 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49ec9f6f25ea477796d52d45b19c5317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.380645 3936 ts_tablet_manager.cc:1403] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.122s user 0.008s sys 0.003s
I20260501 13:58:26.386535 3936 raft_consensus.cc:359] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.369949 3935 ts_tablet_manager.cc:1434] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260501 13:58:26.390353 3935 tablet_bootstrap.cc:492] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:26.396534 3935 tablet_bootstrap.cc:654] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.388670 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 49ec9f6f25ea477796d52d45b19c5317 (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.401448 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49ec9f6f25ea477796d52d45b19c5317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.409101 3936 raft_consensus.cc:385] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.409718 3941 ts_tablet_manager.cc:1929] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.410104 3941 log.cc:1199] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/cf9af59b508b41c0a7bd4efd9bbc369b
I20260501 13:58:26.410719 3941 ts_tablet_manager.cc:1950] T cf9af59b508b41c0a7bd4efd9bbc369b P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:26.409871 3936 raft_consensus.cc:740] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.412920 3936 consensus_queue.cc:260] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.419159 3937 tablet_bootstrap.cc:492] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:26.398864 3942 ts_tablet_manager.cc:1929] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.423540 3942 log.cc:1199] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/cf9af59b508b41c0a7bd4efd9bbc369b
I20260501 13:58:26.424330 3937 tablet_bootstrap.cc:654] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.430166 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet cf9af59b508b41c0a7bd4efd9bbc369b (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
W20260501 13:58:26.431005 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet f2ebffc1b9e849319e05351f926208bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2ebffc1b9e849319e05351f926208bb already in progress: creating tablet
I20260501 13:58:26.436352 3942 ts_tablet_manager.cc:1950] T cf9af59b508b41c0a7bd4efd9bbc369b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:26.439353 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet cf9af59b508b41c0a7bd4efd9bbc369b (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:26.441170 3935 tablet_bootstrap.cc:492] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
W20260501 13:58:26.441041 3213 catalog_manager.cc:4982] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet f2ebffc1b9e849319e05351f926208bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2ebffc1b9e849319e05351f926208bb already in progress: creating tablet
I20260501 13:58:26.441578 3935 ts_tablet_manager.cc:1403] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.002s
I20260501 13:58:26.433452 3936 ts_tablet_manager.cc:1434] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20260501 13:58:26.443694 3937 tablet_bootstrap.cc:492] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:26.444144 3937 ts_tablet_manager.cc:1403] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.004s
I20260501 13:58:26.443921 3935 raft_consensus.cc:359] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.444527 3935 raft_consensus.cc:385] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.444778 3935 raft_consensus.cc:740] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.445441 3935 consensus_queue.cc:260] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.445928 3936 tablet_bootstrap.cc:492] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:26.447456 3935 ts_tablet_manager.cc:1434] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260501 13:58:26.453593 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet f2ebffc1b9e849319e05351f926208bb with delete_type TABLET_DATA_DELETED (replaced by tablet 49ec9f6f25ea477796d52d45b19c5317) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:26.455070 3941 tablet_replica.cc:333] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:26.455711 3941 raft_consensus.cc:2243] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:26.455821 3935 tablet_bootstrap.cc:492] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:26.456143 3941 raft_consensus.cc:2272] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.458642 3941 ts_tablet_manager.cc:1916] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.464926 3937 raft_consensus.cc:359] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.473428 3937 raft_consensus.cc:385] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.473858 3937 raft_consensus.cc:740] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.474594 3937 consensus_queue.cc:260] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.475178 3935 tablet_bootstrap.cc:654] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.476862 3937 ts_tablet_manager.cc:1434] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260501 13:58:26.489542 3224 master_service.cc:954] ReplaceTablet: received request to replace tablet e8ede8efc21a4f2ca821810be62c5e07 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:26.492374 3936 tablet_bootstrap.cc:654] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.505573 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet e8ede8efc21a4f2ca821810be62c5e07 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:26.513799 3936 tablet_bootstrap.cc:492] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:26.514240 3936 ts_tablet_manager.cc:1403] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.000s
I20260501 13:58:26.515795 3941 ts_tablet_manager.cc:1929] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.516160 3941 log.cc:1199] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/f2ebffc1b9e849319e05351f926208bb
I20260501 13:58:26.516755 3941 ts_tablet_manager.cc:1950] T f2ebffc1b9e849319e05351f926208bb P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:26.516798 3936 raft_consensus.cc:359] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.517421 3936 raft_consensus.cc:385] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.517689 3936 raft_consensus.cc:740] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.519170 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet f2ebffc1b9e849319e05351f926208bb (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:26.527488 3224 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet e8ede8efc21a4f2ca821810be62c5e07
I20260501 13:58:26.529349 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet e8ede8efc21a4f2ca821810be62c5e07 with delete_type TABLET_DATA_DELETED (replaced by tablet 36c990cdf8084c5980e0c6ed4a6da2ec) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:26.530251 3941 tablet_replica.cc:333] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:26.530128 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet e8ede8efc21a4f2ca821810be62c5e07 with delete_type TABLET_DATA_DELETED (replaced by tablet 36c990cdf8084c5980e0c6ed4a6da2ec) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:26.530848 3942 tablet_replica.cc:333] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:26.531090 3941 raft_consensus.cc:2243] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:26.531371 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet e8ede8efc21a4f2ca821810be62c5e07 with delete_type TABLET_DATA_DELETED (replaced by tablet 36c990cdf8084c5980e0c6ed4a6da2ec) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:26.531514 3942 raft_consensus.cc:2243] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:26.532050 3943 tablet_replica.cc:333] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:26.532719 3943 raft_consensus.cc:2243] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:26.531939 3942 raft_consensus.cc:2272] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.533166 3943 raft_consensus.cc:2272] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.531699 3941 raft_consensus.cc:2272] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.535727 3943 ts_tablet_manager.cc:1916] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.536473 3941 ts_tablet_manager.cc:1916] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.536990 3224 catalog_manager.cc:6741] ReplaceTablet: tablet e8ede8efc21a4f2ca821810be62c5e07 deleted and replaced by tablet 36c990cdf8084c5980e0c6ed4a6da2ec
I20260501 13:58:26.540279 3935 tablet_bootstrap.cc:492] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:26.540683 3935 ts_tablet_manager.cc:1403] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.093s user 0.015s sys 0.010s
I20260501 13:58:26.543090 3935 raft_consensus.cc:359] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.543692 3935 raft_consensus.cc:385] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.543939 3935 raft_consensus.cc:740] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.548559 3942 ts_tablet_manager.cc:1916] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.544497 3935 consensus_queue.cc:260] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.550122 3941 ts_tablet_manager.cc:1929] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.550889 3941 log.cc:1199] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/e8ede8efc21a4f2ca821810be62c5e07
I20260501 13:58:26.554955 3941 ts_tablet_manager.cc:1950] T e8ede8efc21a4f2ca821810be62c5e07 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:26.556109 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 36c990cdf8084c5980e0c6ed4a6da2ec (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.556032 3935 ts_tablet_manager.cc:1434] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.015s user 0.002s sys 0.005s
I20260501 13:58:26.558974 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 36c990cdf8084c5980e0c6ed4a6da2ec (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.560003 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c990cdf8084c5980e0c6ed4a6da2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.561475 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet e8ede8efc21a4f2ca821810be62c5e07 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:26.564133 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c990cdf8084c5980e0c6ed4a6da2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.518294 3936 consensus_queue.cc:260] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.586483 3936 ts_tablet_manager.cc:1434] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.072s user 0.005s sys 0.000s
I20260501 13:58:26.587194 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 36c990cdf8084c5980e0c6ed4a6da2ec (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.594399 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c990cdf8084c5980e0c6ed4a6da2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.595074 3936 tablet_bootstrap.cc:492] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:26.598930 3935 tablet_bootstrap.cc:492] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:26.581653 3942 ts_tablet_manager.cc:1929] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.599618 3942 log.cc:1199] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/e8ede8efc21a4f2ca821810be62c5e07
I20260501 13:58:26.600741 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet f2ebffc1b9e849319e05351f926208bb with delete_type TABLET_DATA_DELETED (replaced by tablet 49ec9f6f25ea477796d52d45b19c5317) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:26.605702 3936 tablet_bootstrap.cc:654] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.604677 3935 tablet_bootstrap.cc:654] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.614152 3937 tablet_bootstrap.cc:492] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:26.620319 3937 tablet_bootstrap.cc:654] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:26.620437 3943 ts_tablet_manager.cc:1929] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.620805 3943 log.cc:1199] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/e8ede8efc21a4f2ca821810be62c5e07
I20260501 13:58:26.621413 3943 ts_tablet_manager.cc:1950] T e8ede8efc21a4f2ca821810be62c5e07 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:26.623930 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet e8ede8efc21a4f2ca821810be62c5e07 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:26.625936 3935 tablet_bootstrap.cc:492] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:26.626776 3935 ts_tablet_manager.cc:1403] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.000s
I20260501 13:58:26.629285 3935 raft_consensus.cc:359] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.630402 3935 raft_consensus.cc:385] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.630681 3935 raft_consensus.cc:740] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.631588 3937 tablet_bootstrap.cc:492] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:26.631424 3935 consensus_queue.cc:260] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.632103 3937 ts_tablet_manager.cc:1403] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.002s
I20260501 13:58:26.639890 3935 ts_tablet_manager.cc:1434] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260501 13:58:26.641382 3937 raft_consensus.cc:359] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.600756 3942 ts_tablet_manager.cc:1950] T e8ede8efc21a4f2ca821810be62c5e07 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:26.644469 3942 tablet_replica.cc:333] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:26.645190 3942 raft_consensus.cc:2243] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:26.645306 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet e8ede8efc21a4f2ca821810be62c5e07 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:26.645778 3942 raft_consensus.cc:2272] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.647912 3937 raft_consensus.cc:385] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.648241 3937 raft_consensus.cc:740] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:26.650787 3942 ts_tablet_manager.cc:1916] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.653242 3937 consensus_queue.cc:260] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.656558 3945 raft_consensus.cc:493] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:26.656980 3945 raft_consensus.cc:515] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.657541 3937 ts_tablet_manager.cc:1434] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.019s user 0.004s sys 0.001s
I20260501 13:58:26.660678 3945 leader_election.cc:290] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:26.665241 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c990cdf8084c5980e0c6ed4a6da2ec" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:26.666523 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c990cdf8084c5980e0c6ed4a6da2ec" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:26.667106 3515 raft_consensus.cc:2468] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:26.668478 3757 leader_election.cc:304] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:26.669399 3945 raft_consensus.cc:2804] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:26.669792 3945 raft_consensus.cc:493] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:26.670162 3945 raft_consensus.cc:3060] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:58:26.674183 3756 leader_election.cc:343] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:26.677872 3945 raft_consensus.cc:515] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.680787 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c990cdf8084c5980e0c6ed4a6da2ec" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:26.681332 3515 raft_consensus.cc:3060] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:26.695080 3515 raft_consensus.cc:2468] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:26.696432 3757 leader_election.cc:304] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:26.697440 3993 raft_consensus.cc:2804] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:26.709025 3993 raft_consensus.cc:697] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:26.710083 3993 consensus_queue.cc:237] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.715852 3945 leader_election.cc:290] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:26.718189 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c990cdf8084c5980e0c6ed4a6da2ec" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:26.664783 3936 tablet_bootstrap.cc:492] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
W20260501 13:58:26.719499 3756 leader_election.cc:343] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:26.720337 3936 ts_tablet_manager.cc:1403] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.000s
I20260501 13:58:26.723269 3936 raft_consensus.cc:359] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.732995 3224 catalog_manager.cc:5671] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } }
I20260501 13:58:26.735738 3936 raft_consensus.cc:385] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:26.736074 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet d341bb06f3b24efa84f6b84751cb4459 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:26.736059 3936 raft_consensus.cc:740] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
W20260501 13:58:26.746522 3384 hms_client.cc:295] Time spent alter HMS table: real 1.488s user 0.002s sys 0.000s
I20260501 13:58:26.747287 3226 catalog_manager.cc:3502] renamed table default.table_27ad4801c45e4f2da414165d40bf4ae4 [id=c8b517f0258c4d35a40759e478d314ee] in HMS: new name default.table_e3c7989801224024a8debb2eda1686e8
I20260501 13:58:26.755131 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet d341bb06f3b24efa84f6b84751cb4459 from {username='slave'} at 127.0.0.1:50472
W20260501 13:58:26.762538 3225 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b07edfe247554299af96b76fee9a5b4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:26.762205 3222 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:26.774276 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet d341bb06f3b24efa84f6b84751cb4459
I20260501 13:58:26.775760 3227 catalog_manager.cc:6741] ReplaceTablet: tablet d341bb06f3b24efa84f6b84751cb4459 deleted and replaced by tablet e57b93e5dadb4e5cb5077bf79bab3758
I20260501 13:58:26.777762 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet d341bb06f3b24efa84f6b84751cb4459 with delete_type TABLET_DATA_DELETED (replaced by tablet e57b93e5dadb4e5cb5077bf79bab3758) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:26.778573 3941 tablet_replica.cc:333] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:26.780038 3941 raft_consensus.cc:2243] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:26.780435 3941 raft_consensus.cc:2272] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.781930 3941 ts_tablet_manager.cc:1916] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.795706 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet d341bb06f3b24efa84f6b84751cb4459 with delete_type TABLET_DATA_DELETED (replaced by tablet e57b93e5dadb4e5cb5077bf79bab3758) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:26.796855 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet d341bb06f3b24efa84f6b84751cb4459 with delete_type TABLET_DATA_DELETED (replaced by tablet e57b93e5dadb4e5cb5077bf79bab3758) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:26.797626 3943 tablet_replica.cc:333] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:26.798254 3943 raft_consensus.cc:2243] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:26.799015 3943 pending_rounds.cc:70] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: Trying to abort 1 pending ops.
I20260501 13:58:26.799289 3943 pending_rounds.cc:77] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229439743971328 op_type: NO_OP noop_request { }
I20260501 13:58:26.799703 3943 raft_consensus.cc:2889] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:26.800051 3943 raft_consensus.cc:2272] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:26.802688 3947 raft_consensus.cc:493] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:26.803150 3947 raft_consensus.cc:515] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.805657 3943 ts_tablet_manager.cc:1916] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:26.807602 3947 leader_election.cc:290] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:26.809177 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ec9f6f25ea477796d52d45b19c5317" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:26.809456 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ec9f6f25ea477796d52d45b19c5317" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:26.810372 3822 raft_consensus.cc:2468] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:26.811861 3448 leader_election.cc:304] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:26.815935 3947 raft_consensus.cc:2804] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:26.816329 3947 raft_consensus.cc:493] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:26.816635 3947 raft_consensus.cc:3060] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:58:26.841070 3387 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e57b93e5dadb4e5cb5077bf79bab3758 has no consensus state
W20260501 13:58:26.843868 3219 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1f445634a0fc4676a2e7bf208b0ea1de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:26.847545 3220 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ed51787806aa40069d50f787cc85ae8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:26.849797 3947 raft_consensus.cc:515] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.853200 3226 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50472 (request call id 39) took 2705 ms. Trace:
I20260501 13:58:26.853567 3226 rpcz_store.cc:276] 0501 13:58:24.147454 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:24.147593 (+ 139us) service_pool.cc:224] Handling call
0501 13:58:25.102922 (+955329us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:26.850656 (+1747734us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:26.852155 (+ 1499us) catalog_manager.cc:3725] Apply alter schema
0501 13:58:26.853109 (+ 954us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":2482,"Hive Metastore.run_cpu_time_us":2599,"Hive Metastore.run_wall_time_us":1640575}
I20260501 13:58:26.745680 3936 consensus_queue.cc:260] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.855865 3950 raft_consensus.cc:493] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:26.856343 3950 raft_consensus.cc:515] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.857322 3674 raft_consensus.cc:2468] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:26.860925 3226 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50472:
table { table_name: "default.Table_e3c7989801224024a8debb2eda1686e8" } modify_external_catalogs: true
I20260501 13:58:26.861763 3226 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50472:
table { table_name: "default.Table_e3c7989801224024a8debb2eda1686e8" } modify_external_catalogs: true
I20260501 13:58:26.863500 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ec9f6f25ea477796d52d45b19c5317" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:26.863162 3950 leader_election.cc:290] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:26.868269 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ec9f6f25ea477796d52d45b19c5317" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:26.868968 3822 raft_consensus.cc:2468] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:26.870795 3608 leader_election.cc:304] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:26.872900 3950 raft_consensus.cc:2804] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:26.873299 3950 raft_consensus.cc:493] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:26.873622 3950 raft_consensus.cc:3060] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:26.891127 3950 raft_consensus.cc:515] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:26.894177 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ec9f6f25ea477796d52d45b19c5317" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:26.894153 3514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ec9f6f25ea477796d52d45b19c5317" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:26.894774 3822 raft_consensus.cc:3060] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:58:26.898670 3214 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e57b93e5dadb4e5cb5077bf79bab3758 has no consensus state
I20260501 13:58:26.902837 3495 tablet_service.cc:1511] Processing CreateTablet for tablet e57b93e5dadb4e5cb5077bf79bab3758 (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.903865 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57b93e5dadb4e5cb5077bf79bab3758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.924404 3652 tablet_service.cc:1511] Processing CreateTablet for tablet e57b93e5dadb4e5cb5077bf79bab3758 (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.925597 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57b93e5dadb4e5cb5077bf79bab3758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.926115 3802 tablet_service.cc:1511] Processing CreateTablet for tablet e57b93e5dadb4e5cb5077bf79bab3758 (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:26.927103 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57b93e5dadb4e5cb5077bf79bab3758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:26.956085 3515 raft_consensus.cc:2393] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in current term 1: Already voted for candidate 9154033cea374a10a5225c53b8bcf122 in this term.
I20260501 13:58:26.957085 3947 leader_election.cc:290] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:26.958438 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ec9f6f25ea477796d52d45b19c5317" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:26.959707 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ec9f6f25ea477796d52d45b19c5317" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:26.960487 3674 raft_consensus.cc:2393] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9154033cea374a10a5225c53b8bcf122 in current term 1: Already voted for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in this term.
I20260501 13:58:26.961721 3448 leader_election.cc:304] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:26.963397 3947 raft_consensus.cc:2749] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260501 13:58:26.966269 3936 ts_tablet_manager.cc:1434] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.245s user 0.008s sys 0.000s
I20260501 13:58:26.967856 3936 tablet_bootstrap.cc:492] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:26.977449 3942 ts_tablet_manager.cc:1929] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.983274 3935 tablet_bootstrap.cc:492] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:26.985849 3224 master_service.cc:954] ReplaceTablet: received request to replace tablet 36c990cdf8084c5980e0c6ed4a6da2ec from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:26.988407 3935 tablet_bootstrap.cc:654] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:26.010779 3952 debug-util.cc:398] Leaking SignalData structure 0x7b080002ecc0 after lost signal to thread 3914
W20260501 13:58:26.975148 3914 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 2.371s user 0.161s sys 0.305s
W20260501 13:58:26.992208 3914 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 2.388s user 0.162s sys 0.305s
W20260501 13:58:24.625831 3954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:58:26.993857 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet d341bb06f3b24efa84f6b84751cb4459 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:26.995260 3822 raft_consensus.cc:2468] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:26.996552 3941 ts_tablet_manager.cc:1929] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:26.996939 3941 log.cc:1199] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/d341bb06f3b24efa84f6b84751cb4459
I20260501 13:58:26.997552 3941 ts_tablet_manager.cc:1950] T d341bb06f3b24efa84f6b84751cb4459 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:26.999993 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet d341bb06f3b24efa84f6b84751cb4459 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:26.996693 3608 leader_election.cc:304] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:27.004627 3943 ts_tablet_manager.cc:1929] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:27.004837 4008 raft_consensus.cc:2804] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:26.997917 3950 leader_election.cc:290] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:27.008232 3943 log.cc:1199] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/d341bb06f3b24efa84f6b84751cb4459
I20260501 13:58:27.009104 3943 ts_tablet_manager.cc:1950] T d341bb06f3b24efa84f6b84751cb4459 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:27.013305 4008 raft_consensus.cc:697] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:27.014305 4008 consensus_queue.cc:237] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.015640 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet d341bb06f3b24efa84f6b84751cb4459 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
W20260501 13:58:27.016255 4011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:58:27.018433 3942 log.cc:1199] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/f2ebffc1b9e849319e05351f926208bb
I20260501 13:58:27.019205 3942 ts_tablet_manager.cc:1950] T f2ebffc1b9e849319e05351f926208bb P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:27.021862 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet f2ebffc1b9e849319e05351f926208bb (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:27.003953 3936 tablet_bootstrap.cc:654] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.025455 3914 server_base.cc:1061] running on GCE node
I20260501 13:58:27.025982 3942 tablet_replica.cc:333] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:27.026624 3942 raft_consensus.cc:2243] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.026906 3914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:58:27.027089 3942 raft_consensus.cc:2272] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.029796 3942 ts_tablet_manager.cc:1916] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.031023 3224 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 36c990cdf8084c5980e0c6ed4a6da2ec
I20260501 13:58:27.034897 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 36c990cdf8084c5980e0c6ed4a6da2ec with delete_type TABLET_DATA_DELETED (replaced by tablet c50108d8bcfa4106a3965026627b3f12) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:27.037075 3224 catalog_manager.cc:6741] ReplaceTablet: tablet 36c990cdf8084c5980e0c6ed4a6da2ec deleted and replaced by tablet c50108d8bcfa4106a3965026627b3f12
I20260501 13:58:27.037178 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 36c990cdf8084c5980e0c6ed4a6da2ec with delete_type TABLET_DATA_DELETED (replaced by tablet c50108d8bcfa4106a3965026627b3f12) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:27.037850 3943 tablet_replica.cc:333] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:27.038539 3943 raft_consensus.cc:2243] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:27.038502 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet 36c990cdf8084c5980e0c6ed4a6da2ec with delete_type TABLET_DATA_DELETED (replaced by tablet c50108d8bcfa4106a3965026627b3f12) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:27.039183 3943 pending_rounds.cc:70] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: Trying to abort 1 pending ops.
I20260501 13:58:27.039397 3943 pending_rounds.cc:77] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229441894690816 op_type: NO_OP noop_request { }
I20260501 13:58:27.039736 3943 raft_consensus.cc:2889] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:27.040037 3943 raft_consensus.cc:2272] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.039710 3221 catalog_manager.cc:5671] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:27.042003 3941 tablet_replica.cc:333] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:27.042593 3943 ts_tablet_manager.cc:1916] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.042690 3941 raft_consensus.cc:2243] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.043130 3941 raft_consensus.cc:2272] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.045091 3941 ts_tablet_manager.cc:1916] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.064603 3936 tablet_bootstrap.cc:492] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:27.065176 3936 ts_tablet_manager.cc:1403] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.098s user 0.006s sys 0.005s
I20260501 13:58:27.065666 3943 ts_tablet_manager.cc:1929] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:27.066025 3943 log.cc:1199] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/36c990cdf8084c5980e0c6ed4a6da2ec
I20260501 13:58:27.066779 3943 ts_tablet_manager.cc:1950] T 36c990cdf8084c5980e0c6ed4a6da2ec P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:27.068720 3936 raft_consensus.cc:359] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.070366 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 36c990cdf8084c5980e0c6ed4a6da2ec (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:27.070864 3936 raft_consensus.cc:385] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.072192 3914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 13:58:27.072580 3936 raft_consensus.cc:740] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.073450 3936 consensus_queue.cc:260] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.074417 3914 hybrid_clock.cc:648] HybridClock initialized: now 1777643907074355 us; error 50 us; skew 500 ppm
I20260501 13:58:27.075549 3914 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:58:27.078228 3936 ts_tablet_manager.cc:1434] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.013s user 0.004s sys 0.001s
I20260501 13:58:27.088989 3936 tablet_bootstrap.cc:492] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:27.094412 3936 tablet_bootstrap.cc:654] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.096192 3935 tablet_bootstrap.cc:492] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:27.096632 3935 ts_tablet_manager.cc:1403] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.114s user 0.014s sys 0.004s
I20260501 13:58:27.097429 3941 ts_tablet_manager.cc:1929] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.097793 3941 log.cc:1199] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/36c990cdf8084c5980e0c6ed4a6da2ec
I20260501 13:58:27.098423 3941 ts_tablet_manager.cc:1950] T 36c990cdf8084c5980e0c6ed4a6da2ec P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:27.099026 3935 raft_consensus.cc:359] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.099655 3935 raft_consensus.cc:385] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.099907 3935 raft_consensus.cc:740] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.101243 3935 consensus_queue.cc:260] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.103688 3942 ts_tablet_manager.cc:1929] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.104076 3942 log.cc:1199] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/d341bb06f3b24efa84f6b84751cb4459
I20260501 13:58:27.104826 3942 ts_tablet_manager.cc:1950] T d341bb06f3b24efa84f6b84751cb4459 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:27.107879 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 36c990cdf8084c5980e0c6ed4a6da2ec (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:27.108546 3935 ts_tablet_manager.cc:1434] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260501 13:58:27.109938 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet d341bb06f3b24efa84f6b84751cb4459 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:27.111105 3942 tablet_replica.cc:333] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:27.111707 3942 raft_consensus.cc:2243] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.112125 3942 raft_consensus.cc:2272] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.112813 3936 tablet_bootstrap.cc:492] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:27.113255 3936 ts_tablet_manager.cc:1403] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.005s
I20260501 13:58:27.115653 3936 raft_consensus.cc:359] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.117779 3936 raft_consensus.cc:385] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.118074 3936 raft_consensus.cc:740] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.118729 3936 consensus_queue.cc:260] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.117483 3942 ts_tablet_manager.cc:1916] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.128494 3221 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:27.130138 3936 ts_tablet_manager.cc:1434] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.017s user 0.002s sys 0.003s
I20260501 13:58:27.137334 3942 ts_tablet_manager.cc:1929] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.137444 3227 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:27.137923 3942 log.cc:1199] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/36c990cdf8084c5980e0c6ed4a6da2ec
I20260501 13:58:27.138640 3942 ts_tablet_manager.cc:1950] T 36c990cdf8084c5980e0c6ed4a6da2ec P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:27.141012 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 36c990cdf8084c5980e0c6ed4a6da2ec (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:27.145262 3224 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:27.145491 3914 webserver.cc:492] Webserver started at http://127.1.16.188:46681/ using document root <none> and password file <none>
I20260501 13:58:27.147775 3914 fs_manager.cc:362] Metadata directory not provided
I20260501 13:58:27.148249 3914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:58:27.151297 3227 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:27.163686 3495 tablet_service.cc:1511] Processing CreateTablet for tablet c50108d8bcfa4106a3965026627b3f12 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.164798 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50108d8bcfa4106a3965026627b3f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.187036 3652 tablet_service.cc:1511] Processing CreateTablet for tablet c50108d8bcfa4106a3965026627b3f12 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.188345 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50108d8bcfa4106a3965026627b3f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.202936 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 49ec9f6f25ea477796d52d45b19c5317 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:27.223014 3801 tablet_service.cc:1511] Processing CreateTablet for tablet c50108d8bcfa4106a3965026627b3f12 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.224148 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50108d8bcfa4106a3965026627b3f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.234192 4021 tablet_bootstrap.cc:492] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:27.239918 3935 tablet_bootstrap.cc:492] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:27.246274 3935 tablet_bootstrap.cc:654] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.253096 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet c50108d8bcfa4106a3965026627b3f12 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:27.255883 3223 catalog_manager.cc:3502] renamed table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129] in HMS: new name default.table_2840ee60b88a494f9b40b448d78b0000
I20260501 13:58:27.263088 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 49ec9f6f25ea477796d52d45b19c5317
I20260501 13:58:27.265094 3936 tablet_bootstrap.cc:492] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:27.266510 3227 catalog_manager.cc:6741] ReplaceTablet: tablet 49ec9f6f25ea477796d52d45b19c5317 deleted and replaced by tablet 9802591a68dd41e7a60fb3829dcb9858
I20260501 13:58:27.270236 3936 tablet_bootstrap.cc:654] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.270136 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 49ec9f6f25ea477796d52d45b19c5317 with delete_type TABLET_DATA_DELETED (replaced by tablet 9802591a68dd41e7a60fb3829dcb9858) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:27.270826 3943 tablet_replica.cc:333] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:27.271461 3943 raft_consensus.cc:2243] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.271950 3943 raft_consensus.cc:2272] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.272266 4021 tablet_bootstrap.cc:654] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.273705 3943 ts_tablet_manager.cc:1916] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.274771 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 49ec9f6f25ea477796d52d45b19c5317 with delete_type TABLET_DATA_DELETED (replaced by tablet 9802591a68dd41e7a60fb3829dcb9858) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:27.276132 3941 tablet_replica.cc:333] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:27.276805 3941 raft_consensus.cc:2243] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.277267 3941 raft_consensus.cc:2272] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.277637 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet 49ec9f6f25ea477796d52d45b19c5317 with delete_type TABLET_DATA_DELETED (replaced by tablet 9802591a68dd41e7a60fb3829dcb9858) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:27.278352 3942 tablet_replica.cc:333] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:27.278993 3942 raft_consensus.cc:2243] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:27.279716 3942 pending_rounds.cc:70] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: Trying to abort 1 pending ops.
I20260501 13:58:27.279976 3942 pending_rounds.cc:77] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229443142135808 op_type: NO_OP noop_request { }
I20260501 13:58:27.280059 3941 ts_tablet_manager.cc:1916] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.280395 3942 raft_consensus.cc:2889] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:27.280694 3942 raft_consensus.cc:2272] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.283123 3942 ts_tablet_manager.cc:1916] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.287935 3935 tablet_bootstrap.cc:492] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:27.288398 3935 ts_tablet_manager.cc:1403] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.000s
I20260501 13:58:27.291923 3935 raft_consensus.cc:359] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.292559 3935 raft_consensus.cc:385] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.292819 3935 raft_consensus.cc:740] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.302110 3914 fs_manager.cc:714] Time spent opening directory manager: real 0.147s user 0.032s sys 0.043s
I20260501 13:58:27.297428 3935 consensus_queue.cc:260] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.307775 3221 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet c50108d8bcfa4106a3965026627b3f12
I20260501 13:58:27.308306 3936 tablet_bootstrap.cc:492] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:27.308796 3936 ts_tablet_manager.cc:1403] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.001s
I20260501 13:58:27.309243 3221 catalog_manager.cc:6741] ReplaceTablet: tablet c50108d8bcfa4106a3965026627b3f12 deleted and replaced by tablet 40d2f66187da4468af57074f93c19927
I20260501 13:58:27.311374 3936 raft_consensus.cc:359] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.313463 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet c50108d8bcfa4106a3965026627b3f12 with delete_type TABLET_DATA_DELETED (replaced by tablet 40d2f66187da4468af57074f93c19927) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:27.316215 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet c50108d8bcfa4106a3965026627b3f12 with delete_type TABLET_DATA_DELETED (replaced by tablet 40d2f66187da4468af57074f93c19927) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:27.319094 3935 ts_tablet_manager.cc:1434] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.030s user 0.007s sys 0.001s
I20260501 13:58:27.320428 3941 ts_tablet_manager.cc:1929] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.320794 3941 log.cc:1199] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/49ec9f6f25ea477796d52d45b19c5317
I20260501 13:58:27.321420 3941 ts_tablet_manager.cc:1950] T 49ec9f6f25ea477796d52d45b19c5317 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:27.315687 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet c50108d8bcfa4106a3965026627b3f12 with delete_type TABLET_DATA_DELETED (replaced by tablet 40d2f66187da4468af57074f93c19927) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:27.333922 3943 ts_tablet_manager.cc:1929] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.330214 3936 raft_consensus.cc:385] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.334312 3943 log.cc:1199] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/49ec9f6f25ea477796d52d45b19c5317
I20260501 13:58:27.334964 3943 ts_tablet_manager.cc:1950] T 49ec9f6f25ea477796d52d45b19c5317 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:27.334427 3936 raft_consensus.cc:740] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.336007 3936 consensus_queue.cc:260] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.337247 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 49ec9f6f25ea477796d52d45b19c5317 (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:27.343097 3936 ts_tablet_manager.cc:1434] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.034s user 0.004s sys 0.002s
I20260501 13:58:27.338845 4021 tablet_bootstrap.cc:492] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:27.343568 4021 ts_tablet_manager.cc:1403] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.110s user 0.014s sys 0.000s
I20260501 13:58:27.346395 4026 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:27.344132 3942 ts_tablet_manager.cc:1929] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:27.347143 3942 log.cc:1199] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/49ec9f6f25ea477796d52d45b19c5317
I20260501 13:58:27.347940 3942 ts_tablet_manager.cc:1950] T 49ec9f6f25ea477796d52d45b19c5317 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:27.348904 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 9802591a68dd41e7a60fb3829dcb9858 (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.349864 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9802591a68dd41e7a60fb3829dcb9858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.351137 3941 tablet_replica.cc:333] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:27.351078 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 9802591a68dd41e7a60fb3829dcb9858 (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.351773 3941 raft_consensus.cc:2243] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.352108 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9802591a68dd41e7a60fb3829dcb9858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.352185 3941 raft_consensus.cc:2272] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.355072 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 49ec9f6f25ea477796d52d45b19c5317 (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:27.346334 4021 raft_consensus.cc:359] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.355940 4021 raft_consensus.cc:385] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.356245 4021 raft_consensus.cc:740] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.357250 3914 fs_manager.cc:730] Time spent opening block manager: real 0.034s user 0.002s sys 0.001s
I20260501 13:58:27.359103 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 9802591a68dd41e7a60fb3829dcb9858 (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.360152 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9802591a68dd41e7a60fb3829dcb9858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.357712 3914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal
uuid: "6b0f8911dd5d431d9d8d3124a4456262"
format_stamp: "Formatted at 2026-05-01 13:58:13 on dist-test-slave-nqxd"
I20260501 13:58:27.363447 3942 tablet_replica.cc:333] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:27.363782 3914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:58:27.363956 4021 consensus_queue.cc:260] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.365238 3942 raft_consensus.cc:2243] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.371130 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 49ec9f6f25ea477796d52d45b19c5317 (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:27.371750 3942 raft_consensus.cc:2272] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.381902 4021 ts_tablet_manager.cc:1434] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20260501 13:58:27.382639 3941 ts_tablet_manager.cc:1916] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.385047 3942 ts_tablet_manager.cc:1916] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.386616 4021 tablet_bootstrap.cc:492] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:27.399326 3224 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:27.405910 3227 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:27.407342 3494 tablet_service.cc:1511] Processing CreateTablet for tablet 40d2f66187da4468af57074f93c19927 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.408478 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40d2f66187da4468af57074f93c19927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.412987 4021 tablet_bootstrap.cc:654] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.416011 3941 ts_tablet_manager.cc:1929] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.416411 3941 log.cc:1199] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/c50108d8bcfa4106a3965026627b3f12
I20260501 13:58:27.417028 3941 ts_tablet_manager.cc:1950] T c50108d8bcfa4106a3965026627b3f12 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:27.419526 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet c50108d8bcfa4106a3965026627b3f12 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:27.424898 3936 tablet_bootstrap.cc:492] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:27.429428 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 40d2f66187da4468af57074f93c19927 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.430648 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40d2f66187da4468af57074f93c19927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.431404 3942 ts_tablet_manager.cc:1929] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.432528 3942 log.cc:1199] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/c50108d8bcfa4106a3965026627b3f12
I20260501 13:58:27.436272 3935 tablet_bootstrap.cc:492] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:27.434646 3936 tablet_bootstrap.cc:654] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.438092 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 40d2f66187da4468af57074f93c19927 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.439214 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40d2f66187da4468af57074f93c19927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.443856 4021 tablet_bootstrap.cc:492] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:27.436323 3942 ts_tablet_manager.cc:1950] T c50108d8bcfa4106a3965026627b3f12 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:27.444312 4021 ts_tablet_manager.cc:1403] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.000s
I20260501 13:58:27.446870 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet c50108d8bcfa4106a3965026627b3f12 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:27.448479 3935 tablet_bootstrap.cc:654] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.449340 4021 raft_consensus.cc:359] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.449990 4021 raft_consensus.cc:385] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.450254 4021 raft_consensus.cc:740] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
W20260501 13:58:27.451495 3213 catalog_manager.cc:4982] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet c50108d8bcfa4106a3965026627b3f12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c50108d8bcfa4106a3965026627b3f12 already in progress: creating tablet
I20260501 13:58:27.454434 3936 tablet_bootstrap.cc:492] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:27.454864 3936 ts_tablet_manager.cc:1403] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.004s
I20260501 13:58:27.457389 3936 raft_consensus.cc:359] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.457986 3936 raft_consensus.cc:385] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.458225 3936 raft_consensus.cc:740] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.460925 4008 raft_consensus.cc:493] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:27.461354 4008 raft_consensus.cc:515] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.463136 4008 leader_election.cc:290] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:27.463909 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57b93e5dadb4e5cb5077bf79bab3758" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:27.464529 3515 raft_consensus.cc:2468] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:27.466235 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57b93e5dadb4e5cb5077bf79bab3758" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:27.466521 3609 leader_election.cc:304] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:27.466420 4021 consensus_queue.cc:260] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.466859 3822 raft_consensus.cc:2468] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:27.467298 4008 raft_consensus.cc:2804] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:27.467605 4008 raft_consensus.cc:493] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:27.467908 4008 raft_consensus.cc:3060] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:27.470314 3936 consensus_queue.cc:260] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.475121 4021 ts_tablet_manager.cc:1434] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20260501 13:58:27.476066 4021 tablet_bootstrap.cc:492] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:27.476672 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet e57b93e5dadb4e5cb5077bf79bab3758 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:27.481400 3936 ts_tablet_manager.cc:1434] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.026s user 0.005s sys 0.001s
I20260501 13:58:27.482301 3936 tablet_bootstrap.cc:492] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:27.490047 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet c50108d8bcfa4106a3965026627b3f12 with delete_type TABLET_DATA_DELETED (replaced by tablet 40d2f66187da4468af57074f93c19927) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:27.492375 3943 tablet_replica.cc:333] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:27.493054 3943 raft_consensus.cc:2243] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.493515 3943 raft_consensus.cc:2272] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.496119 4008 raft_consensus.cc:515] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.498365 3943 ts_tablet_manager.cc:1916] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.498973 4021 tablet_bootstrap.cc:654] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.500059 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57b93e5dadb4e5cb5077bf79bab3758" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:27.500610 3515 raft_consensus.cc:3060] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:27.507658 3936 tablet_bootstrap.cc:654] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.507128 3515 raft_consensus.cc:2468] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:27.508464 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet e57b93e5dadb4e5cb5077bf79bab3758
I20260501 13:58:27.508780 3609 leader_election.cc:304] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:27.509670 4012 raft_consensus.cc:2804] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:27.509876 3227 catalog_manager.cc:6741] ReplaceTablet: tablet e57b93e5dadb4e5cb5077bf79bab3758 deleted and replaced by tablet c84730af31394c148ac52abf0b93e52e
I20260501 13:58:27.510128 4012 raft_consensus.cc:697] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:27.510949 4012 consensus_queue.cc:237] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.515378 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet e57b93e5dadb4e5cb5077bf79bab3758 with delete_type TABLET_DATA_DELETED (replaced by tablet c84730af31394c148ac52abf0b93e52e) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:27.516067 3941 tablet_replica.cc:333] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:27.516718 3941 raft_consensus.cc:2243] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.517143 3941 raft_consensus.cc:2272] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.519877 3941 ts_tablet_manager.cc:1916] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.525674 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet e57b93e5dadb4e5cb5077bf79bab3758 with delete_type TABLET_DATA_DELETED (replaced by tablet c84730af31394c148ac52abf0b93e52e) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:27.526726 3222 catalog_manager.cc:2174] created HMS catalog entry for table [id=b1334bc9986e4463add9b290ab6294ef]
I20260501 13:58:27.527263 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet e57b93e5dadb4e5cb5077bf79bab3758 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:27.543206 3222 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50472 (request call id 42) took 3195 ms. Trace:
I20260501 13:58:27.543545 3222 rpcz_store.cc:276] 0501 13:58:24.347876 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:24.348058 (+ 182us) service_pool.cc:224] Handling call
0501 13:58:26.763263 (+2415205us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:26.765035 (+ 1772us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:27.526685 (+761650us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:27.542531 (+ 15846us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:27.542839 (+ 308us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:27.543138 (+ 299us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:27.528557 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:27.529024 (+ 467us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:27.529050 (+ 26us) write_op.cc:435] Acquired schema lock
0501 13:58:27.529064 (+ 14us) tablet.cc:662] Decoding operations
0501 13:58:27.529364 (+ 300us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:27.529404 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:27.529430 (+ 26us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:27.529633 (+ 203us) tablet.cc:701] Row locks acquired
0501 13:58:27.529649 (+ 16us) write_op.cc:260] PREPARE: finished
0501 13:58:27.529774 (+ 125us) write_op.cc:270] Start()
0501 13:58:27.529864 (+ 90us) write_op.cc:276] Timestamp: P: 1777643907529722 usec, L: 0
0501 13:58:27.529883 (+ 19us) op_driver.cc:348] REPLICATION: starting
0501 13:58:27.530416 (+ 533us) log.cc:844] Serialized 432 byte log entry
0501 13:58:27.540597 (+ 10181us) op_driver.cc:464] REPLICATION: finished
0501 13:58:27.541076 (+ 479us) write_op.cc:301] APPLY: starting
0501 13:58:27.541130 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:27.541242 (+ 112us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:27.541321 (+ 79us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:27.541720 (+ 399us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:27.541844 (+ 124us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:27.541883 (+ 39us) write_op.cc:312] APPLY: finished
0501 13:58:27.542059 (+ 176us) log.cc:844] Serialized 28 byte log entry
0501 13:58:27.542194 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:27.542260 (+ 66us) write_op.cc:454] Released schema lock
0501 13:58:27.542288 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:27.542528 (+ 240us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:27.542549 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":489842,"Hive Metastore.run_cpu_time_us":600,"Hive Metastore.run_wall_time_us":270587,"child_traces":[["op",{"apply.queue_time_us":329,"apply.run_cpu_time_us":1404,"apply.run_wall_time_us":1404,"num_ops":2,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":2227,"prepare.run_wall_time_us":4065,"raft.queue_time_us":183,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":13,"replication_time_us":10617}]]}
I20260501 13:58:27.549238 4021 tablet_bootstrap.cc:492] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:27.549686 4021 ts_tablet_manager.cc:1403] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.003s
I20260501 13:58:27.551028 3222 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50472:
table { table_name: "default.Table_1d615bc9ac7a4db39b4d4afb4ef38d8a" } new_table_name: "default.Table_4db61a7562964779afbc5ff93b00a5f5" modify_external_catalogs: true
I20260501 13:58:27.552312 4021 raft_consensus.cc:359] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.552939 4021 raft_consensus.cc:385] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.553205 4021 raft_consensus.cc:740] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.557807 3943 ts_tablet_manager.cc:1929] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.558211 3943 log.cc:1199] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/c50108d8bcfa4106a3965026627b3f12
I20260501 13:58:27.565970 3494 tablet_service.cc:1511] Processing CreateTablet for tablet c84730af31394c148ac52abf0b93e52e (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.567070 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84730af31394c148ac52abf0b93e52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.565397 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet e57b93e5dadb4e5cb5077bf79bab3758 with delete_type TABLET_DATA_DELETED (replaced by tablet c84730af31394c148ac52abf0b93e52e) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:27.568002 3942 tablet_replica.cc:333] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:27.572391 4021 consensus_queue.cc:260] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.580556 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57b93e5dadb4e5cb5077bf79bab3758" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:27.581195 3822 raft_consensus.cc:3060] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:27.588333 3802 tablet_service.cc:1511] Processing CreateTablet for tablet c84730af31394c148ac52abf0b93e52e (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.589406 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84730af31394c148ac52abf0b93e52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.590256 4021 ts_tablet_manager.cc:1434] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.040s user 0.003s sys 0.004s
I20260501 13:58:27.591084 3822 raft_consensus.cc:2468] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:27.591349 4021 tablet_bootstrap.cc:492] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:27.602662 3652 tablet_service.cc:1511] Processing CreateTablet for tablet c84730af31394c148ac52abf0b93e52e (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.603837 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84730af31394c148ac52abf0b93e52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.606099 4021 tablet_bootstrap.cc:654] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.606238 3942 raft_consensus.cc:2243] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:27.608421 3942 pending_rounds.cc:70] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: Trying to abort 1 pending ops.
I20260501 13:58:27.608651 3942 pending_rounds.cc:77] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229445184180224 op_type: NO_OP noop_request { }
I20260501 13:58:27.608989 3942 raft_consensus.cc:2889] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:27.609315 3942 raft_consensus.cc:2272] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.615808 4008 leader_election.cc:290] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:27.620237 3936 tablet_bootstrap.cc:492] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
W20260501 13:58:27.620186 4033 consensus_queue.cc:1175] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20260501 13:58:27.620677 3936 ts_tablet_manager.cc:1403] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.139s user 0.008s sys 0.012s
I20260501 13:58:27.623063 3936 raft_consensus.cc:359] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.623648 3936 raft_consensus.cc:385] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.623884 3936 raft_consensus.cc:740] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.624442 3936 consensus_queue.cc:260] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.626080 3942 ts_tablet_manager.cc:1916] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.637318 3943 ts_tablet_manager.cc:1950] T c50108d8bcfa4106a3965026627b3f12 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:27.626097 3936 ts_tablet_manager.cc:1434] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260501 13:58:27.656128 4021 tablet_bootstrap.cc:492] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:27.656584 4021 ts_tablet_manager.cc:1403] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.000s
I20260501 13:58:27.656769 3495 tablet_service.cc:1511] Processing CreateTablet for tablet d2353f7e06b24b64864cd6073b0b58db (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.657822 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2353f7e06b24b64864cd6073b0b58db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.658357 3935 tablet_bootstrap.cc:492] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:27.658762 3935 ts_tablet_manager.cc:1403] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.223s user 0.014s sys 0.000s
I20260501 13:58:27.661046 3935 raft_consensus.cc:359] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.661645 3935 raft_consensus.cc:385] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.661885 3935 raft_consensus.cc:740] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.662755 3914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:58:27.663357 3941 ts_tablet_manager.cc:1929] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.663687 3941 log.cc:1199] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/e57b93e5dadb4e5cb5077bf79bab3758
I20260501 13:58:27.664268 3941 ts_tablet_manager.cc:1950] T e57b93e5dadb4e5cb5077bf79bab3758 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:27.665243 3935 consensus_queue.cc:260] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.666010 3947 raft_consensus.cc:493] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:27.666400 3947 raft_consensus.cc:515] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.668654 3942 ts_tablet_manager.cc:1929] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:27.669018 3942 log.cc:1199] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/e57b93e5dadb4e5cb5077bf79bab3758
I20260501 13:58:27.669663 3942 ts_tablet_manager.cc:1950] T e57b93e5dadb4e5cb5077bf79bab3758 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:27.671355 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9802591a68dd41e7a60fb3829dcb9858" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:27.671512 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9802591a68dd41e7a60fb3829dcb9858" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:27.671940 3822 raft_consensus.cc:2468] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:27.672055 3674 raft_consensus.cc:2468] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:27.672084 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:27.673064 3448 leader_election.cc:304] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:27.672585 3947 leader_election.cc:290] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:27.674124 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:27.683682 3947 raft_consensus.cc:2804] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:27.684088 3947 raft_consensus.cc:493] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:27.665796 3914 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:58:27.684427 3947 raft_consensus.cc:3060] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:27.684958 3914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:58:27.692999 3225 catalog_manager.cc:2174] created HMS catalog entry for table [id=a71004edce1a4d5298f01382612a99ee]
I20260501 13:58:27.717337 3654 tablet_service.cc:1511] Processing CreateTablet for tablet d2353f7e06b24b64864cd6073b0b58db (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.717931 4021 raft_consensus.cc:359] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.718740 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2353f7e06b24b64864cd6073b0b58db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.718691 4021 raft_consensus.cc:385] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.718979 4021 raft_consensus.cc:740] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.719648 4021 consensus_queue.cc:260] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.722378 4021 ts_tablet_manager.cc:1434] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.065s user 0.007s sys 0.000s
I20260501 13:58:27.723487 4021 tablet_bootstrap.cc:492] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:27.729743 4021 tablet_bootstrap.cc:654] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.732439 3225 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50472 (request call id 45) took 3021 ms. Trace:
I20260501 13:58:27.733815 3943 tablet_replica.cc:333] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:27.734656 3943 raft_consensus.cc:2243] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.734977 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet c50108d8bcfa4106a3965026627b3f12 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:27.746605 3943 raft_consensus.cc:2272] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:58:27.747506 3448 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:45857, user_credentials={real_user=slave}} blocked reactor thread for 74493.9us
I20260501 13:58:27.752452 3943 ts_tablet_manager.cc:1916] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.753587 3224 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_aafd3ab3f9f44a74af41aae60cfe40fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:27.753949 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet 40d2f66187da4468af57074f93c19927 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:27.732785 3225 rpcz_store.cc:276] 0501 13:58:24.711358 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:24.712386 (+ 1028us) service_pool.cc:224] Handling call
0501 13:58:24.721794 (+ 9408us) spinlock_profiling.cc:243] Waited 8.37 ms on lock 0x7b5400030554. stack: 00007fc5c69e2c83 00007fc5c69e2b80 00007fc5c6553574 000000000050e094 000000000050e029 00007fc5cda32470 00007fc5cdad131a 00007fc5cf7e546c 00007fc5cf8e0936 00007fc5cd4ed524 00007fc5cd4ed4b2 00007fc5cd4ed3e1 00007fc5cd4ed35c 00007fc5cd4ec612 00007fc5cb7bbbcc 00007fc5cb7bb006
0501 13:58:26.763017 (+2041223us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:26.766813 (+ 3796us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:27.692951 (+926138us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:27.731732 (+ 38781us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:27.732067 (+ 335us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:27.732375 (+ 308us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:27.698245 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:27.698862 (+ 617us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:27.698895 (+ 33us) write_op.cc:435] Acquired schema lock
0501 13:58:27.698911 (+ 16us) tablet.cc:662] Decoding operations
0501 13:58:27.699254 (+ 343us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:27.699299 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:27.699324 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:27.699525 (+ 201us) tablet.cc:701] Row locks acquired
0501 13:58:27.699539 (+ 14us) write_op.cc:260] PREPARE: finished
0501 13:58:27.699667 (+ 128us) write_op.cc:270] Start()
0501 13:58:27.699754 (+ 87us) write_op.cc:276] Timestamp: P: 1777643907699613 usec, L: 0
0501 13:58:27.699770 (+ 16us) op_driver.cc:348] REPLICATION: starting
0501 13:58:27.700345 (+ 575us) log.cc:844] Serialized 432 byte log entry
0501 13:58:27.727726 (+ 27381us) op_driver.cc:464] REPLICATION: finished
0501 13:58:27.729242 (+ 1516us) write_op.cc:301] APPLY: starting
0501 13:58:27.729308 (+ 66us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:27.729408 (+ 100us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:27.729502 (+ 94us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:27.729868 (+ 366us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:27.730002 (+ 134us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:27.730048 (+ 46us) write_op.cc:312] APPLY: finished
0501 13:58:27.730244 (+ 196us) log.cc:844] Serialized 28 byte log entry
0501 13:58:27.730387 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:27.730457 (+ 70us) write_op.cc:454] Released schema lock
0501 13:58:27.730509 (+ 52us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":758752,"Hive Metastore.run_cpu_time_us":661,"Hive Metastore.run_wall_time_us":166408,"spinlock_wait_cycles":19262336,"child_traces":[["op",{"apply.queue_time_us":1323,"apply.run_cpu_time_us":1505,"apply.run_wall_time_us":1505,"num_ops":2,"prepare.queue_time_us":2579,"prepare.run_cpu_time_us":2597,"prepare.run_wall_time_us":41246,"raft.queue_time_us":16759,"raft.run_cpu_time_us":803,"raft.run_wall_time_us":816,"replication_time_us":27853}]]}
I20260501 13:58:27.760716 3225 master_service.cc:954] ReplaceTablet: received request to replace tablet 9802591a68dd41e7a60fb3829dcb9858 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:27.745239 3802 tablet_service.cc:1511] Processing CreateTablet for tablet d2353f7e06b24b64864cd6073b0b58db (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.767619 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2353f7e06b24b64864cd6073b0b58db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.781095 3221 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 40d2f66187da4468af57074f93c19927
I20260501 13:58:27.781569 3935 ts_tablet_manager.cc:1434] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.123s user 0.021s sys 0.022s
I20260501 13:58:27.782549 3221 catalog_manager.cc:6741] ReplaceTablet: tablet 40d2f66187da4468af57074f93c19927 deleted and replaced by tablet 0d06ec6cc444404f9ec2eb3c91a48528
I20260501 13:58:27.782578 3935 tablet_bootstrap.cc:492] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:27.786837 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 40d2f66187da4468af57074f93c19927 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d06ec6cc444404f9ec2eb3c91a48528) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:27.787408 3653 tablet_service.cc:1558] Processing DeleteTablet for tablet 40d2f66187da4468af57074f93c19927 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d06ec6cc444404f9ec2eb3c91a48528) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:27.788133 3942 tablet_replica.cc:333] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:27.788765 3942 raft_consensus.cc:2243] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.789224 3942 raft_consensus.cc:2272] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.793147 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet 40d2f66187da4468af57074f93c19927 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d06ec6cc444404f9ec2eb3c91a48528) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:27.797812 3942 ts_tablet_manager.cc:1916] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:58:27.798781 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 40d2f66187da4468af57074f93c19927 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40d2f66187da4468af57074f93c19927 already in progress: creating tablet
I20260501 13:58:27.806824 3947 raft_consensus.cc:515] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.819614 3947 leader_election.cc:290] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:27.820482 3936 tablet_bootstrap.cc:492] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:27.823414 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9802591a68dd41e7a60fb3829dcb9858" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:27.823614 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9802591a68dd41e7a60fb3829dcb9858" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:27.824023 3674 raft_consensus.cc:3060] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:27.824215 3822 raft_consensus.cc:3060] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:27.828512 3936 tablet_bootstrap.cc:654] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.830814 3674 raft_consensus.cc:2468] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:27.830904 3822 raft_consensus.cc:2468] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:27.833462 3935 tablet_bootstrap.cc:654] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.833693 3448 leader_election.cc:304] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:27.842175 3225 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 9802591a68dd41e7a60fb3829dcb9858
I20260501 13:58:27.834286 3943 ts_tablet_manager.cc:1929] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.842978 3943 log.cc:1199] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/e57b93e5dadb4e5cb5077bf79bab3758
I20260501 13:58:27.843648 3225 catalog_manager.cc:6741] ReplaceTablet: tablet 9802591a68dd41e7a60fb3829dcb9858 deleted and replaced by tablet 7b34fd361c8346938ce68b6e0e726d8d
I20260501 13:58:27.843681 3943 ts_tablet_manager.cc:1950] T e57b93e5dadb4e5cb5077bf79bab3758 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:27.845070 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 9802591a68dd41e7a60fb3829dcb9858 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b34fd361c8346938ce68b6e0e726d8d) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:27.845949 3943 tablet_replica.cc:333] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:27.846176 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:27.852276 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 9802591a68dd41e7a60fb3829dcb9858 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b34fd361c8346938ce68b6e0e726d8d) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:27.852432 3943 raft_consensus.cc:2243] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.852914 3943 raft_consensus.cc:2272] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.853014 3941 tablet_replica.cc:333] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:27.853646 3941 raft_consensus.cc:2243] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.854100 3941 raft_consensus.cc:2272] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.856271 3941 ts_tablet_manager.cc:1916] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.858178 3943 ts_tablet_manager.cc:1916] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.863524 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 9802591a68dd41e7a60fb3829dcb9858 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b34fd361c8346938ce68b6e0e726d8d) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:27.870317 4021 tablet_bootstrap.cc:492] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:27.870851 4021 ts_tablet_manager.cc:1403] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.148s user 0.020s sys 0.058s
I20260501 13:58:27.873775 4021 raft_consensus.cc:359] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.874485 4021 raft_consensus.cc:385] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.874774 4021 raft_consensus.cc:740] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.877409 3941 ts_tablet_manager.cc:1929] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.877767 3941 log.cc:1199] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/9802591a68dd41e7a60fb3829dcb9858
I20260501 13:58:27.881198 3936 tablet_bootstrap.cc:492] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:27.881623 3936 ts_tablet_manager.cc:1403] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.000s
I20260501 13:58:27.884275 3936 raft_consensus.cc:359] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.885968 3936 raft_consensus.cc:385] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.891290 3936 raft_consensus.cc:740] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.891955 3936 consensus_queue.cc:260] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.882895 4021 consensus_queue.cc:260] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.903643 3225 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:27.904217 4021 ts_tablet_manager.cc:1434] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
W20260501 13:58:27.905269 3225 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet c84730af31394c148ac52abf0b93e52e consensus state has no leader
I20260501 13:58:27.906965 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 0d06ec6cc444404f9ec2eb3c91a48528 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.907982 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d06ec6cc444404f9ec2eb3c91a48528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.912140 3943 ts_tablet_manager.cc:1929] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.917677 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 0d06ec6cc444404f9ec2eb3c91a48528 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.919652 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d06ec6cc444404f9ec2eb3c91a48528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.919641 3494 tablet_service.cc:1511] Processing CreateTablet for tablet e5368e2d21694ad9bf15dce464b52f1e (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.920771 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5368e2d21694ad9bf15dce464b52f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.922206 3225 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:27.928965 3654 tablet_service.cc:1511] Processing CreateTablet for tablet e5368e2d21694ad9bf15dce464b52f1e (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.929967 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5368e2d21694ad9bf15dce464b52f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.939994 3942 ts_tablet_manager.cc:1929] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:27.940398 3942 log.cc:1199] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/40d2f66187da4468af57074f93c19927
I20260501 13:58:27.940997 3942 ts_tablet_manager.cc:1950] T 40d2f66187da4468af57074f93c19927 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:27.925643 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 0d06ec6cc444404f9ec2eb3c91a48528 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.941500 3943 log.cc:1199] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/40d2f66187da4468af57074f93c19927
I20260501 13:58:27.943414 3936 ts_tablet_manager.cc:1434] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.062s user 0.007s sys 0.000s
I20260501 13:58:27.941978 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d06ec6cc444404f9ec2eb3c91a48528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.933590 4021 tablet_bootstrap.cc:492] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
W20260501 13:58:27.950825 3214 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c84730af31394c148ac52abf0b93e52e consensus state has no leader
I20260501 13:58:27.945245 3936 tablet_bootstrap.cc:492] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:27.955171 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 40d2f66187da4468af57074f93c19927 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:27.956056 3942 tablet_replica.cc:333] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:27.954756 3943 ts_tablet_manager.cc:1950] T 40d2f66187da4468af57074f93c19927 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:27.956717 3942 raft_consensus.cc:2243] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.957181 3942 raft_consensus.cc:2272] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.959332 3935 tablet_bootstrap.cc:492] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:27.959733 3935 ts_tablet_manager.cc:1403] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.177s user 0.005s sys 0.004s
I20260501 13:58:27.962934 3225 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_06e2e16b792549e68d2e01ce3e3ce8b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:27.964092 3801 tablet_service.cc:1511] Processing CreateTablet for tablet e5368e2d21694ad9bf15dce464b52f1e (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:27.965175 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5368e2d21694ad9bf15dce464b52f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:27.966228 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 40d2f66187da4468af57074f93c19927 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:27.878402 3941 ts_tablet_manager.cc:1950] T 9802591a68dd41e7a60fb3829dcb9858 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:27.962114 3935 raft_consensus.cc:359] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.967823 3935 raft_consensus.cc:385] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:27.968102 3935 raft_consensus.cc:740] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:27.972765 4021 tablet_bootstrap.cc:654] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.968158 3943 tablet_replica.cc:333] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:27.975899 3943 raft_consensus.cc:2243] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:27.976416 3943 raft_consensus.cc:2272] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:27.978523 3943 ts_tablet_manager.cc:1916] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:27.980283 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 9802591a68dd41e7a60fb3829dcb9858 (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:27.981240 3935 consensus_queue.cc:260] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:27.988754 3942 ts_tablet_manager.cc:1916] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.001750 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 40d2f66187da4468af57074f93c19927 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d06ec6cc444404f9ec2eb3c91a48528) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:28.003299 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet e5368e2d21694ad9bf15dce464b52f1e from {username='slave'} at 127.0.0.1:50472
W20260501 13:58:28.004390 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 40d2f66187da4468af57074f93c19927 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40d2f66187da4468af57074f93c19927 already in progress: creating tablet
I20260501 13:58:28.006183 3936 tablet_bootstrap.cc:654] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:27.982932 3935 ts_tablet_manager.cc:1434] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260501 13:58:28.007728 3935 tablet_bootstrap.cc:492] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:28.009382 4008 raft_consensus.cc:493] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:28.009804 4008 raft_consensus.cc:515] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.011662 4008 leader_election.cc:290] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:28.013549 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2353f7e06b24b64864cd6073b0b58db" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:28.015167 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2353f7e06b24b64864cd6073b0b58db" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
W20260501 13:58:28.016551 3608 leader_election.cc:343] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:58:28.017606 3609 leader_election.cc:343] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:28.017988 3609 leader_election.cc:304] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:28.018965 4008 raft_consensus.cc:2749] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:28.021431 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 40d2f66187da4468af57074f93c19927 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d06ec6cc444404f9ec2eb3c91a48528) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:28.022253 3941 tablet_replica.cc:333] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:28.022903 3941 raft_consensus.cc:2243] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.023370 3941 raft_consensus.cc:2272] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.025350 3941 ts_tablet_manager.cc:1916] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.032788 4021 tablet_bootstrap.cc:492] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:28.033257 4021 ts_tablet_manager.cc:1403] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.100s user 0.008s sys 0.002s
I20260501 13:58:28.034523 3936 tablet_bootstrap.cc:492] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:28.034927 3936 ts_tablet_manager.cc:1403] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.091s user 0.010s sys 0.000s
I20260501 13:58:28.036010 3943 ts_tablet_manager.cc:1929] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.036990 3943 log.cc:1199] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/9802591a68dd41e7a60fb3829dcb9858
I20260501 13:58:28.037621 3943 ts_tablet_manager.cc:1950] T 9802591a68dd41e7a60fb3829dcb9858 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:28.051517 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 9802591a68dd41e7a60fb3829dcb9858 (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:28.051950 4021 raft_consensus.cc:359] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.052621 4021 raft_consensus.cc:385] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.052907 4021 raft_consensus.cc:740] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.053620 4021 consensus_queue.cc:260] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.057072 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet d2353f7e06b24b64864cd6073b0b58db from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:28.067358 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet e5368e2d21694ad9bf15dce464b52f1e
I20260501 13:58:28.074178 3221 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet d2353f7e06b24b64864cd6073b0b58db
I20260501 13:58:28.075716 3227 catalog_manager.cc:6741] ReplaceTablet: tablet e5368e2d21694ad9bf15dce464b52f1e deleted and replaced by tablet 7ac3e55605994b10899afcba0ba6abe9
I20260501 13:58:28.078711 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet d2353f7e06b24b64864cd6073b0b58db with delete_type TABLET_DATA_DELETED (replaced by tablet 8aa8986f92c4462fbc159f17cc282a0b) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:28.080060 3221 catalog_manager.cc:6741] ReplaceTablet: tablet d2353f7e06b24b64864cd6073b0b58db deleted and replaced by tablet 8aa8986f92c4462fbc159f17cc282a0b
I20260501 13:58:28.081729 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet e5368e2d21694ad9bf15dce464b52f1e with delete_type TABLET_DATA_DELETED (replaced by tablet 7ac3e55605994b10899afcba0ba6abe9) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:28.083114 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet e5368e2d21694ad9bf15dce464b52f1e with delete_type TABLET_DATA_DELETED (replaced by tablet 7ac3e55605994b10899afcba0ba6abe9) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:28.083740 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet d2353f7e06b24b64864cd6073b0b58db with delete_type TABLET_DATA_DELETED (replaced by tablet 8aa8986f92c4462fbc159f17cc282a0b) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:28.085135 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 7b34fd361c8346938ce68b6e0e726d8d (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.086158 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b34fd361c8346938ce68b6e0e726d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.082427 4021 ts_tablet_manager.cc:1434] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.049s user 0.002s sys 0.005s
I20260501 13:58:28.082372 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet e5368e2d21694ad9bf15dce464b52f1e with delete_type TABLET_DATA_DELETED (replaced by tablet 7ac3e55605994b10899afcba0ba6abe9) from {username='slave'} at 127.0.0.1:46676
W20260501 13:58:28.088981 3213 catalog_manager.cc:4982] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet e5368e2d21694ad9bf15dce464b52f1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5368e2d21694ad9bf15dce464b52f1e already in progress: creating tablet
I20260501 13:58:28.082845 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet d2353f7e06b24b64864cd6073b0b58db with delete_type TABLET_DATA_DELETED (replaced by tablet 8aa8986f92c4462fbc159f17cc282a0b) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:28.092128 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 7b34fd361c8346938ce68b6e0e726d8d (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.093117 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b34fd361c8346938ce68b6e0e726d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.095268 3935 tablet_bootstrap.cc:654] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:28.096539 3213 catalog_manager.cc:4982] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet d2353f7e06b24b64864cd6073b0b58db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2353f7e06b24b64864cd6073b0b58db already in progress: creating tablet
I20260501 13:58:28.098680 3936 raft_consensus.cc:359] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.101377 3801 tablet_service.cc:1511] Processing CreateTablet for tablet 7b34fd361c8346938ce68b6e0e726d8d (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.102710 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b34fd361c8346938ce68b6e0e726d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.103206 3942 ts_tablet_manager.cc:1929] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.103585 3942 log.cc:1199] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/9802591a68dd41e7a60fb3829dcb9858
I20260501 13:58:28.104254 3942 ts_tablet_manager.cc:1950] T 9802591a68dd41e7a60fb3829dcb9858 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:28.099326 3936 raft_consensus.cc:385] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.106324 3936 raft_consensus.cc:740] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.107323 4021 tablet_bootstrap.cc:492] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:28.107173 3936 consensus_queue.cc:260] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.113302 3221 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:28.113664 4021 tablet_bootstrap.cc:654] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.118350 4008 raft_consensus.cc:493] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:28.118829 4008 raft_consensus.cc:515] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.120632 4008 leader_election.cc:290] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:28.122808 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet d2353f7e06b24b64864cd6073b0b58db with delete_type TABLET_DATA_DELETED (replaced by tablet 8aa8986f92c4462fbc159f17cc282a0b) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:28.123482 3943 tablet_replica.cc:333] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:28.123920 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c84730af31394c148ac52abf0b93e52e" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:28.124079 3943 raft_consensus.cc:2243] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.124243 3936 ts_tablet_manager.cc:1434] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.089s user 0.006s sys 0.000s
I20260501 13:58:28.124506 3943 raft_consensus.cc:2272] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260501 13:58:28.125622 3609 leader_election.cc:343] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:28.126626 3943 ts_tablet_manager.cc:1916] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.129596 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c84730af31394c148ac52abf0b93e52e" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:28.130185 3822 raft_consensus.cc:2468] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:28.129820 3941 ts_tablet_manager.cc:1929] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.130443 3221 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:28.132542 3608 leader_election.cc:304] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:28.133185 3941 log.cc:1199] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/40d2f66187da4468af57074f93c19927
I20260501 13:58:28.133908 3941 ts_tablet_manager.cc:1950] T 40d2f66187da4468af57074f93c19927 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:28.135191 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 9802591a68dd41e7a60fb3829dcb9858 (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:28.135180 4008 raft_consensus.cc:2804] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:28.136222 4008 raft_consensus.cc:493] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:28.136554 4008 raft_consensus.cc:3060] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:28.142236 3942 tablet_replica.cc:333] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
W20260501 13:58:28.145766 3213 catalog_manager.cc:4982] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet e5368e2d21694ad9bf15dce464b52f1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5368e2d21694ad9bf15dce464b52f1e already in progress: creating tablet
I20260501 13:58:28.158231 3942 raft_consensus.cc:2243] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.158274 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 40d2f66187da4468af57074f93c19927 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:28.158747 3942 raft_consensus.cc:2272] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.134339 3936 tablet_bootstrap.cc:492] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:28.142989 4008 raft_consensus.cc:515] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.163836 4008 leader_election.cc:290] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:28.164943 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c84730af31394c148ac52abf0b93e52e" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:28.165491 3822 raft_consensus.cc:3060] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:28.166296 3935 tablet_bootstrap.cc:492] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:28.166733 3935 ts_tablet_manager.cc:1403] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.159s user 0.009s sys 0.001s
I20260501 13:58:28.168823 3935 raft_consensus.cc:359] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.169293 3935 raft_consensus.cc:385] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.169493 3935 raft_consensus.cc:740] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.169955 3935 consensus_queue.cc:260] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.170991 3943 ts_tablet_manager.cc:1929] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.171362 3943 log.cc:1199] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/d2353f7e06b24b64864cd6073b0b58db
I20260501 13:58:28.171948 3943 ts_tablet_manager.cc:1950] T d2353f7e06b24b64864cd6073b0b58db P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:28.172120 3822 raft_consensus.cc:2468] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:28.173228 3608 leader_election.cc:304] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:28.174203 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet d2353f7e06b24b64864cd6073b0b58db (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:28.174641 4008 raft_consensus.cc:2804] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:28.175879 3935 ts_tablet_manager.cc:1434] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.009s user 0.001s sys 0.004s
I20260501 13:58:28.174269 3942 ts_tablet_manager.cc:1916] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.176761 4021 tablet_bootstrap.cc:492] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:28.177937 4021 ts_tablet_manager.cc:1403] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.007s
I20260501 13:58:28.178932 3936 tablet_bootstrap.cc:654] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.181407 4021 raft_consensus.cc:359] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.182009 4021 raft_consensus.cc:385] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.183946 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c84730af31394c148ac52abf0b93e52e" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:28.184481 3515 raft_consensus.cc:3060] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:58:28.187122 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet d2353f7e06b24b64864cd6073b0b58db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2353f7e06b24b64864cd6073b0b58db already in progress: creating tablet
W20260501 13:58:28.187813 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet e5368e2d21694ad9bf15dce464b52f1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5368e2d21694ad9bf15dce464b52f1e already in progress: creating tablet
I20260501 13:58:28.188369 3935 tablet_bootstrap.cc:492] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:28.182281 4021 raft_consensus.cc:740] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.193295 3515 raft_consensus.cc:2468] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:28.193192 4021 consensus_queue.cc:260] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.195211 4021 ts_tablet_manager.cc:1434] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260501 13:58:28.206918 4008 raft_consensus.cc:697] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:28.208226 4021 tablet_bootstrap.cc:492] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:28.208298 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet e5368e2d21694ad9bf15dce464b52f1e with delete_type TABLET_DATA_DELETED (replaced by tablet 7ac3e55605994b10899afcba0ba6abe9) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:28.208528 4008 consensus_queue.cc:237] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.213657 3943 tablet_replica.cc:333] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:28.214332 3943 raft_consensus.cc:2243] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.214864 3943 raft_consensus.cc:2272] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.216431 3221 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:28.218577 3943 ts_tablet_manager.cc:1916] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.219821 3935 tablet_bootstrap.cc:654] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.223042 3936 tablet_bootstrap.cc:492] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:28.223457 3936 ts_tablet_manager.cc:1403] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.089s user 0.014s sys 0.000s
I20260501 13:58:28.223832 3227 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:28.225848 3936 raft_consensus.cc:359] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.228677 3936 raft_consensus.cc:385] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.228960 3936 raft_consensus.cc:740] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.229565 3936 consensus_queue.cc:260] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.230233 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 8aa8986f92c4462fbc159f17cc282a0b (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.231323 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa8986f92c4462fbc159f17cc282a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.231331 3936 ts_tablet_manager.cc:1434] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260501 13:58:28.236858 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 8aa8986f92c4462fbc159f17cc282a0b (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.237803 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa8986f92c4462fbc159f17cc282a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.240927 3942 ts_tablet_manager.cc:1929] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.248984 3935 tablet_bootstrap.cc:492] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:28.249414 3935 ts_tablet_manager.cc:1403] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.000s
I20260501 13:58:28.248674 3221 catalog_manager.cc:5671] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:28.254995 3221 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:28.255738 4021 tablet_bootstrap.cc:654] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.255648 3935 raft_consensus.cc:359] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.256256 3935 raft_consensus.cc:385] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.243788 3936 tablet_bootstrap.cc:492] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:28.261948 3936 tablet_bootstrap.cc:654] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.241277 3942 log.cc:1199] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/d2353f7e06b24b64864cd6073b0b58db
I20260501 13:58:28.263324 3942 ts_tablet_manager.cc:1950] T d2353f7e06b24b64864cd6073b0b58db P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:28.256507 3935 raft_consensus.cc:740] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.264356 3935 consensus_queue.cc:260] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.272158 3935 ts_tablet_manager.cc:1434] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260501 13:58:28.276050 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet d2353f7e06b24b64864cd6073b0b58db with delete_type TABLET_DATA_DELETED (replaced by tablet 8aa8986f92c4462fbc159f17cc282a0b) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:28.276750 3935 tablet_bootstrap.cc:492] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:28.277040 4021 tablet_bootstrap.cc:492] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:28.277459 4021 ts_tablet_manager.cc:1403] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.003s
I20260501 13:58:28.276747 3941 tablet_replica.cc:333] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:28.280241 3941 raft_consensus.cc:2243] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.280712 3941 raft_consensus.cc:2272] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.281193 3801 tablet_service.cc:1511] Processing CreateTablet for tablet 8aa8986f92c4462fbc159f17cc282a0b (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.282269 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa8986f92c4462fbc159f17cc282a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.295077 3941 ts_tablet_manager.cc:1916] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.296334 3494 tablet_service.cc:1511] Processing CreateTablet for tablet 7ac3e55605994b10899afcba0ba6abe9 (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.297580 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ac3e55605994b10899afcba0ba6abe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.299081 4021 raft_consensus.cc:359] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.302292 4021 raft_consensus.cc:385] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.302634 4021 raft_consensus.cc:740] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.303366 4021 consensus_queue.cc:260] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.310760 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet e5368e2d21694ad9bf15dce464b52f1e from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:28.312101 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet c84730af31394c148ac52abf0b93e52e from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:28.325951 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 7ac3e55605994b10899afcba0ba6abe9 (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.326622 3935 tablet_bootstrap.cc:654] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.334267 3941 ts_tablet_manager.cc:1929] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.334681 3941 log.cc:1199] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/d2353f7e06b24b64864cd6073b0b58db
I20260501 13:58:28.335318 3941 ts_tablet_manager.cc:1950] T d2353f7e06b24b64864cd6073b0b58db P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:28.338021 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 7ac3e55605994b10899afcba0ba6abe9 (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.339063 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ac3e55605994b10899afcba0ba6abe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.342722 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet d2353f7e06b24b64864cd6073b0b58db (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:28.331724 3943 ts_tablet_manager.cc:1929] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.350963 3943 log.cc:1199] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/e5368e2d21694ad9bf15dce464b52f1e
I20260501 13:58:28.351589 3943 ts_tablet_manager.cc:1950] T e5368e2d21694ad9bf15dce464b52f1e P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:28.353256 3935 tablet_bootstrap.cc:492] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:28.353924 3935 ts_tablet_manager.cc:1403] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.003s
I20260501 13:58:28.354810 4021 ts_tablet_manager.cc:1434] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.077s user 0.006s sys 0.002s
I20260501 13:58:28.360684 3935 raft_consensus.cc:359] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.361351 3935 raft_consensus.cc:385] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.361593 3935 raft_consensus.cc:740] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.364699 4021 tablet_bootstrap.cc:492] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:28.364336 3935 consensus_queue.cc:260] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.366339 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet e5368e2d21694ad9bf15dce464b52f1e (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:28.374547 3935 ts_tablet_manager.cc:1434] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260501 13:58:28.375926 4021 tablet_bootstrap.cc:654] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.396896 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet c84730af31394c148ac52abf0b93e52e
I20260501 13:58:28.398252 3227 catalog_manager.cc:6741] ReplaceTablet: tablet c84730af31394c148ac52abf0b93e52e deleted and replaced by tablet 90fc78d9f46f409f91cf4193afe4398d
I20260501 13:58:28.402184 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet c84730af31394c148ac52abf0b93e52e with delete_type TABLET_DATA_DELETED (replaced by tablet 90fc78d9f46f409f91cf4193afe4398d) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:28.402262 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet c84730af31394c148ac52abf0b93e52e with delete_type TABLET_DATA_DELETED (replaced by tablet 90fc78d9f46f409f91cf4193afe4398d) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:28.402292 3653 tablet_service.cc:1558] Processing DeleteTablet for tablet c84730af31394c148ac52abf0b93e52e with delete_type TABLET_DATA_DELETED (replaced by tablet 90fc78d9f46f409f91cf4193afe4398d) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:28.402901 3943 tablet_replica.cc:333] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:28.403015 3941 tablet_replica.cc:333] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:28.403561 3943 raft_consensus.cc:2243] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.403661 3941 raft_consensus.cc:2243] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.404006 3943 raft_consensus.cc:2272] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.404078 3941 raft_consensus.cc:2272] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.405601 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet e5368e2d21694ad9bf15dce464b52f1e with delete_type TABLET_DATA_DELETED (replaced by tablet 7ac3e55605994b10899afcba0ba6abe9) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:28.406072 3943 ts_tablet_manager.cc:1916] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.406405 3941 ts_tablet_manager.cc:1916] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.439548 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ac3e55605994b10899afcba0ba6abe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.452196 3942 tablet_replica.cc:333] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:28.452836 3942 raft_consensus.cc:2243] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:28.453488 3942 pending_rounds.cc:70] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: Trying to abort 1 pending ops.
I20260501 13:58:28.453712 3942 pending_rounds.cc:77] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229448056131584 op_type: NO_OP noop_request { }
I20260501 13:58:28.454032 3942 raft_consensus.cc:2889] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:28.454910 3942 raft_consensus.cc:2272] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.454819 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet d2353f7e06b24b64864cd6073b0b58db (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:28.457450 3942 ts_tablet_manager.cc:1916] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.468847 3936 tablet_bootstrap.cc:492] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:28.469281 3936 ts_tablet_manager.cc:1403] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.226s user 0.036s sys 0.042s
I20260501 13:58:28.471700 3936 raft_consensus.cc:359] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.472323 3936 raft_consensus.cc:385] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.472574 3936 raft_consensus.cc:740] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.472806 3943 ts_tablet_manager.cc:1929] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.473165 3943 log.cc:1199] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/c84730af31394c148ac52abf0b93e52e
I20260501 13:58:28.473162 3936 consensus_queue.cc:260] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.473795 3943 ts_tablet_manager.cc:1950] T c84730af31394c148ac52abf0b93e52e P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:28.474087 4021 tablet_bootstrap.cc:492] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:28.474529 4021 ts_tablet_manager.cc:1403] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.110s user 0.012s sys 0.003s
I20260501 13:58:28.476145 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:28.477711 4021 raft_consensus.cc:359] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.478277 3942 ts_tablet_manager.cc:1929] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:28.479192 3935 tablet_bootstrap.cc:492] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:28.482888 3941 ts_tablet_manager.cc:1929] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.478386 4021 raft_consensus.cc:385] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.483719 4021 raft_consensus.cc:740] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.484496 4021 consensus_queue.cc:260] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.489290 3936 ts_tablet_manager.cc:1434] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260501 13:58:28.490232 3936 tablet_bootstrap.cc:492] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:28.478658 3942 log.cc:1199] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/c84730af31394c148ac52abf0b93e52e
I20260501 13:58:28.495914 3942 ts_tablet_manager.cc:1950] T c84730af31394c148ac52abf0b93e52e P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:28.499490 3936 tablet_bootstrap.cc:654] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.500741 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:28.511365 3936 tablet_bootstrap.cc:492] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:28.511771 3936 ts_tablet_manager.cc:1403] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.003s
I20260501 13:58:28.514058 3936 raft_consensus.cc:359] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.514730 3936 raft_consensus.cc:385] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.514986 3936 raft_consensus.cc:740] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.517086 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet e5368e2d21694ad9bf15dce464b52f1e with delete_type TABLET_DATA_DELETED (replaced by tablet 7ac3e55605994b10899afcba0ba6abe9) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:28.518784 4021 ts_tablet_manager.cc:1434] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.044s user 0.005s sys 0.000s
I20260501 13:58:28.519512 3942 tablet_replica.cc:333] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:28.520087 3942 raft_consensus.cc:2243] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.520502 3942 raft_consensus.cc:2272] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.526680 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet e5368e2d21694ad9bf15dce464b52f1e from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:28.528038 3942 ts_tablet_manager.cc:1916] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.515553 3936 consensus_queue.cc:260] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.543417 3936 ts_tablet_manager.cc:1434] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.031s user 0.003s sys 0.003s
I20260501 13:58:28.544378 3936 tablet_bootstrap.cc:492] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:28.484742 3935 tablet_bootstrap.cc:654] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.541364 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 90fc78d9f46f409f91cf4193afe4398d (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.550892 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90fc78d9f46f409f91cf4193afe4398d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.557818 4027 raft_consensus.cc:493] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:28.558274 4027 raft_consensus.cc:515] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.560499 4027 leader_election.cc:290] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:28.562227 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b34fd361c8346938ce68b6e0e726d8d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
W20260501 13:58:28.563705 3757 leader_election.cc:343] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:28.568325 4021 tablet_bootstrap.cc:492] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:28.548101 3941 log.cc:1199] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/c84730af31394c148ac52abf0b93e52e
I20260501 13:58:28.569209 3941 ts_tablet_manager.cc:1950] T c84730af31394c148ac52abf0b93e52e P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:28.575999 4021 tablet_bootstrap.cc:654] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.578913 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:28.585426 3941 tablet_replica.cc:333] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:28.586228 3941 raft_consensus.cc:2243] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.586781 3941 raft_consensus.cc:2272] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.589416 3941 ts_tablet_manager.cc:1916] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.582733 3801 tablet_service.cc:1511] Processing CreateTablet for tablet 90fc78d9f46f409f91cf4193afe4398d (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.590950 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90fc78d9f46f409f91cf4193afe4398d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.602106 3942 ts_tablet_manager.cc:1929] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.602553 3942 log.cc:1199] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/e5368e2d21694ad9bf15dce464b52f1e
I20260501 13:58:28.603173 3942 ts_tablet_manager.cc:1950] T e5368e2d21694ad9bf15dce464b52f1e P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:28.611908 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 8aa8986f92c4462fbc159f17cc282a0b from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:28.614367 3936 tablet_bootstrap.cc:654] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.618363 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b34fd361c8346938ce68b6e0e726d8d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:28.619079 3674 raft_consensus.cc:2468] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
W20260501 13:58:28.620811 3213 catalog_manager.cc:5075] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): alter failed for tablet c84730af31394c148ac52abf0b93e52e (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]),no further retry: Not found: Tablet not found: c84730af31394c148ac52abf0b93e52e
I20260501 13:58:28.622539 3756 leader_election.cc:304] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:28.624239 4027 raft_consensus.cc:2804] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:28.624553 4027 raft_consensus.cc:493] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:28.624895 4027 raft_consensus.cc:3060] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:28.633421 4027 raft_consensus.cc:515] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.636013 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b34fd361c8346938ce68b6e0e726d8d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
W20260501 13:58:28.637260 3757 leader_election.cc:343] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:28.638401 4027 leader_election.cc:290] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:28.639236 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 8aa8986f92c4462fbc159f17cc282a0b
I20260501 13:58:28.640898 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aa8986f92c4462fbc159f17cc282a0b with delete_type TABLET_DATA_DELETED (replaced by tablet 36f21071de974647b03747d15e5f8ccc) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:28.641510 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b34fd361c8346938ce68b6e0e726d8d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:28.641876 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet e5368e2d21694ad9bf15dce464b52f1e (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:28.642058 3674 raft_consensus.cc:3060] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:28.642169 3227 catalog_manager.cc:6741] ReplaceTablet: tablet 8aa8986f92c4462fbc159f17cc282a0b deleted and replaced by tablet 36f21071de974647b03747d15e5f8ccc
I20260501 13:58:28.643416 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aa8986f92c4462fbc159f17cc282a0b with delete_type TABLET_DATA_DELETED (replaced by tablet 36f21071de974647b03747d15e5f8ccc) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:28.644444 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aa8986f92c4462fbc159f17cc282a0b with delete_type TABLET_DATA_DELETED (replaced by tablet 36f21071de974647b03747d15e5f8ccc) from {username='slave'} at 127.0.0.1:49092
W20260501 13:58:28.644704 3213 catalog_manager.cc:4982] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet 8aa8986f92c4462fbc159f17cc282a0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aa8986f92c4462fbc159f17cc282a0b already in progress: creating tablet
W20260501 13:58:28.646872 3213 catalog_manager.cc:4982] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet 8aa8986f92c4462fbc159f17cc282a0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aa8986f92c4462fbc159f17cc282a0b already in progress: creating tablet
I20260501 13:58:28.647132 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 90fc78d9f46f409f91cf4193afe4398d (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.648319 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90fc78d9f46f409f91cf4193afe4398d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.650985 3674 raft_consensus.cc:2468] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:28.652313 3756 leader_election.cc:304] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:28.653402 4027 raft_consensus.cc:2804] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:28.660684 3935 tablet_bootstrap.cc:492] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:28.661139 3935 ts_tablet_manager.cc:1403] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.182s user 0.006s sys 0.017s
I20260501 13:58:28.668169 3935 raft_consensus.cc:359] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.668812 3935 raft_consensus.cc:385] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.669068 3935 raft_consensus.cc:740] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.669656 3935 consensus_queue.cc:260] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.670488 4027 raft_consensus.cc:697] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:28.670683 4021 tablet_bootstrap.cc:492] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:28.671078 4021 ts_tablet_manager.cc:1403] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.000s
I20260501 13:58:28.673349 4027 consensus_queue.cc:237] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.673365 4021 raft_consensus.cc:359] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.675472 4021 raft_consensus.cc:385] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.675719 4021 raft_consensus.cc:740] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.676313 4021 consensus_queue.cc:260] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.679672 4100 raft_consensus.cc:493] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:28.680111 4100 raft_consensus.cc:515] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.682343 4021 ts_tablet_manager.cc:1434] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260501 13:58:28.682902 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ac3e55605994b10899afcba0ba6abe9" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:28.683524 4021 tablet_bootstrap.cc:492] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
W20260501 13:58:28.684197 3757 leader_election.cc:343] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:28.687247 3936 tablet_bootstrap.cc:492] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:28.690205 3936 ts_tablet_manager.cc:1403] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.146s user 0.007s sys 0.008s
I20260501 13:58:28.697394 3935 ts_tablet_manager.cc:1434] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.036s user 0.006s sys 0.001s
I20260501 13:58:28.699779 4100 leader_election.cc:290] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:28.702410 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ac3e55605994b10899afcba0ba6abe9" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:28.713898 3936 raft_consensus.cc:359] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.714609 3936 raft_consensus.cc:385] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.714892 3936 raft_consensus.cc:740] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.715458 3936 consensus_queue.cc:260] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.717226 3936 ts_tablet_manager.cc:1434] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20260501 13:58:28.718112 3936 tablet_bootstrap.cc:492] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:28.724807 3941 ts_tablet_manager.cc:1929] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.725206 3941 log.cc:1199] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/e5368e2d21694ad9bf15dce464b52f1e
I20260501 13:58:28.725826 3941 ts_tablet_manager.cc:1950] T e5368e2d21694ad9bf15dce464b52f1e P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:28.728302 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet e5368e2d21694ad9bf15dce464b52f1e (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
W20260501 13:58:28.738277 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 8aa8986f92c4462fbc159f17cc282a0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aa8986f92c4462fbc159f17cc282a0b already in progress: creating tablet
I20260501 13:58:28.739883 3936 tablet_bootstrap.cc:654] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.748175 3936 tablet_bootstrap.cc:492] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:28.749426 3221 catalog_manager.cc:5671] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } }
I20260501 13:58:28.753185 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 36f21071de974647b03747d15e5f8ccc (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.754159 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f21071de974647b03747d15e5f8ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.754628 4100 raft_consensus.cc:493] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:28.755064 4100 raft_consensus.cc:515] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.756855 4100 leader_election.cc:290] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:28.761365 3801 tablet_service.cc:1511] Processing CreateTablet for tablet 36f21071de974647b03747d15e5f8ccc (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.762508 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f21071de974647b03747d15e5f8ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.766633 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 7ac3e55605994b10899afcba0ba6abe9 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:28.780534 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d06ec6cc444404f9ec2eb3c91a48528" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:28.781160 3515 raft_consensus.cc:2468] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:28.781925 3935 tablet_bootstrap.cc:492] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:28.793804 3757 leader_election.cc:304] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:28.796944 4027 raft_consensus.cc:2804] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:28.797323 4027 raft_consensus.cc:493] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:28.797617 4027 raft_consensus.cc:3060] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:28.817783 3914 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.188:38253
I20260501 13:58:28.817845 4114 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.188:38253 every 8 connection(s)
W20260501 13:58:28.825224 3756 leader_election.cc:343] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:28.825688 3756 leader_election.cc:304] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:28.826309 4021 tablet_bootstrap.cc:654] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.827215 4100 raft_consensus.cc:2749] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:28.831071 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 3914
I20260501 13:58:28.831398 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 7ac3e55605994b10899afcba0ba6abe9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7152aff22d5744d8919a82aa2e5f3798) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:28.822352 3914 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-2/data/info.pb
I20260501 13:58:28.832173 3943 tablet_replica.cc:333] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:28.832798 3943 raft_consensus.cc:2243] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.833240 3943 raft_consensus.cc:2272] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.835120 3943 ts_tablet_manager.cc:1916] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.838426 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 7ac3e55605994b10899afcba0ba6abe9
I20260501 13:58:28.840128 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 7ac3e55605994b10899afcba0ba6abe9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7152aff22d5744d8919a82aa2e5f3798) from {username='slave'} at 127.0.0.1:58038
W20260501 13:58:28.841825 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 7ac3e55605994b10899afcba0ba6abe9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ac3e55605994b10899afcba0ba6abe9 already in progress: creating tablet
I20260501 13:58:28.845027 3227 catalog_manager.cc:6741] ReplaceTablet: tablet 7ac3e55605994b10899afcba0ba6abe9 deleted and replaced by tablet 7152aff22d5744d8919a82aa2e5f3798
I20260501 13:58:28.847199 4027 raft_consensus.cc:515] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.849107 4027 leader_election.cc:290] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:28.851380 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet 0d06ec6cc444404f9ec2eb3c91a48528 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:28.852531 3935 tablet_bootstrap.cc:654] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.859117 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d06ec6cc444404f9ec2eb3c91a48528" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:28.859767 3515 raft_consensus.cc:3060] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:28.871833 3935 tablet_bootstrap.cc:492] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:28.872264 3935 ts_tablet_manager.cc:1403] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.000s
I20260501 13:58:28.866336 3515 raft_consensus.cc:2468] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:28.874963 3757 leader_election.cc:304] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:28.875842 4027 raft_consensus.cc:2804] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:28.876374 4021 tablet_bootstrap.cc:492] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:28.877246 4021 ts_tablet_manager.cc:1403] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.194s user 0.006s sys 0.004s
I20260501 13:58:28.882299 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d06ec6cc444404f9ec2eb3c91a48528" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:28.882656 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 36f21071de974647b03747d15e5f8ccc (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:28.883646 3673 raft_consensus.cc:3060] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:28.884631 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f21071de974647b03747d15e5f8ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:28.885656 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aa8986f92c4462fbc159f17cc282a0b with delete_type TABLET_DATA_DELETED (replaced by tablet 36f21071de974647b03747d15e5f8ccc) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:28.887066 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d06ec6cc444404f9ec2eb3c91a48528" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:28.893157 3673 raft_consensus.cc:2468] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:28.880724 3935 raft_consensus.cc:359] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.895489 3935 raft_consensus.cc:385] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.896174 3935 raft_consensus.cc:740] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.897552 3935 consensus_queue.cc:260] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.748576 3936 ts_tablet_manager.cc:1403] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.001s
I20260501 13:58:28.898967 3942 tablet_replica.cc:333] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:28.877507 4027 raft_consensus.cc:697] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:28.900134 3942 raft_consensus.cc:2243] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.900679 3942 raft_consensus.cc:2272] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.900555 4027 consensus_queue.cc:237] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.902737 3936 raft_consensus.cc:359] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.903515 3936 raft_consensus.cc:385] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.903853 3942 ts_tablet_manager.cc:1916] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:28.903802 3936 raft_consensus.cc:740] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.904647 3936 consensus_queue.cc:260] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.908977 3936 ts_tablet_manager.cc:1434] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260501 13:58:28.910035 3936 tablet_bootstrap.cc:492] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:28.906010 4021 raft_consensus.cc:359] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.911216 4021 raft_consensus.cc:385] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:28.911460 4021 raft_consensus.cc:740] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:28.921308 3935 ts_tablet_manager.cc:1434] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20260501 13:58:28.922842 3936 tablet_bootstrap.cc:654] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.923743 3935 tablet_bootstrap.cc:492] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:28.925974 4021 consensus_queue.cc:260] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.929219 4021 ts_tablet_manager.cc:1434] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.025s user 0.004s sys 0.000s
I20260501 13:58:28.929991 4021 tablet_bootstrap.cc:492] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:28.930653 4116 sys_catalog.cc:263] Verifying existing consensus state
I20260501 13:58:28.932164 3935 tablet_bootstrap.cc:654] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.951957 4027 raft_consensus.cc:493] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:28.952499 4027 raft_consensus.cc:515] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:28.956549 4027 leader_election.cc:290] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:28.956789 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aa8986f92c4462fbc159f17cc282a0b" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
W20260501 13:58:28.958176 3756 leader_election.cc:343] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: Tablet not RUNNING: STOPPED
I20260501 13:58:28.960223 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aa8986f92c4462fbc159f17cc282a0b" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
W20260501 13:58:28.961403 3757 leader_election.cc:343] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:28.961840 3757 leader_election.cc:304] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:28.963128 4027 raft_consensus.cc:2749] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:28.972180 4021 tablet_bootstrap.cc:654] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:28.978781 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet 7ac3e55605994b10899afcba0ba6abe9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7152aff22d5744d8919a82aa2e5f3798) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:28.983628 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 7ac3e55605994b10899afcba0ba6abe9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7152aff22d5744d8919a82aa2e5f3798) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:28.987607 3221 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 0d06ec6cc444404f9ec2eb3c91a48528
I20260501 13:58:28.988938 3221 catalog_manager.cc:6741] ReplaceTablet: tablet 0d06ec6cc444404f9ec2eb3c91a48528 deleted and replaced by tablet be144bbf949844539848011a9963ac2d
I20260501 13:58:28.990438 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 0d06ec6cc444404f9ec2eb3c91a48528 with delete_type TABLET_DATA_DELETED (replaced by tablet be144bbf949844539848011a9963ac2d) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:28.989624 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet 0d06ec6cc444404f9ec2eb3c91a48528 with delete_type TABLET_DATA_DELETED (replaced by tablet be144bbf949844539848011a9963ac2d) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:28.992712 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 0d06ec6cc444404f9ec2eb3c91a48528 with delete_type TABLET_DATA_DELETED (replaced by tablet be144bbf949844539848011a9963ac2d) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:28.993391 3941 tablet_replica.cc:333] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:28.993969 3941 raft_consensus.cc:2243] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:28.994378 3941 raft_consensus.cc:2272] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:28.994545 3942 ts_tablet_manager.cc:1929] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.994915 3942 log.cc:1199] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/8aa8986f92c4462fbc159f17cc282a0b
I20260501 13:58:28.995661 3942 ts_tablet_manager.cc:1950] T 8aa8986f92c4462fbc159f17cc282a0b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:28.989233 3943 ts_tablet_manager.cc:1929] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:28.998003 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 8aa8986f92c4462fbc159f17cc282a0b (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:28.998282 3943 log.cc:1199] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/7ac3e55605994b10899afcba0ba6abe9
I20260501 13:58:29.001577 3943 ts_tablet_manager.cc:1950] T 7ac3e55605994b10899afcba0ba6abe9 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:29.002712 3941 ts_tablet_manager.cc:1916] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.005264 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 7ac3e55605994b10899afcba0ba6abe9 (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:29.005857 4021 tablet_bootstrap.cc:492] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:29.006246 4021 ts_tablet_manager.cc:1403] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.003s
I20260501 13:58:29.009486 4021 raft_consensus.cc:359] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.010067 4021 raft_consensus.cc:385] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.010290 4021 raft_consensus.cc:740] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.013082 3942 tablet_replica.cc:333] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:29.013773 3942 raft_consensus.cc:2243] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.014212 3942 raft_consensus.cc:2272] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.016569 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aa8986f92c4462fbc159f17cc282a0b with delete_type TABLET_DATA_DELETED (replaced by tablet 36f21071de974647b03747d15e5f8ccc) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:29.017469 3942 ts_tablet_manager.cc:1916] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.017273 4021 consensus_queue.cc:260] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:29.019853 3213 catalog_manager.cc:4969] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet 7ac3e55605994b10899afcba0ba6abe9 because the tablet was not found. No further retry: Not found: Tablet not found: 7ac3e55605994b10899afcba0ba6abe9
I20260501 13:58:29.027603 3943 tablet_replica.cc:333] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:29.027571 4116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Bootstrap starting.
I20260501 13:58:29.031229 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aa8986f92c4462fbc159f17cc282a0b with delete_type TABLET_DATA_DELETED (replaced by tablet 36f21071de974647b03747d15e5f8ccc) from {username='slave'} at 127.0.0.1:49092
W20260501 13:58:29.038959 3214 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e57b93e5dadb4e5cb5077bf79bab3758 consensus state has no leader
I20260501 13:58:29.039294 3943 raft_consensus.cc:2243] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:29.031616 4021 ts_tablet_manager.cc:1434] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.025s user 0.004s sys 0.000s
I20260501 13:58:29.046561 3935 tablet_bootstrap.cc:492] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:29.047079 3935 ts_tablet_manager.cc:1403] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.124s user 0.017s sys 0.013s
I20260501 13:58:29.052479 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 7152aff22d5744d8919a82aa2e5f3798 (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.053455 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7152aff22d5744d8919a82aa2e5f3798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.058621 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 36f21071de974647b03747d15e5f8ccc from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:29.047181 3943 pending_rounds.cc:70] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: Trying to abort 1 pending ops.
I20260501 13:58:29.058992 3943 pending_rounds.cc:77] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229450866335744 op_type: NO_OP noop_request { }
I20260501 13:58:29.059352 3943 raft_consensus.cc:2889] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:29.059671 3943 raft_consensus.cc:2272] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.060441 4021 tablet_bootstrap.cc:492] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:29.062160 3935 raft_consensus.cc:359] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.067073 4021 tablet_bootstrap.cc:654] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.062832 3935 raft_consensus.cc:385] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.070544 3935 raft_consensus.cc:740] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.071206 3935 consensus_queue.cc:260] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.077106 3935 ts_tablet_manager.cc:1434] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.030s user 0.003s sys 0.002s
I20260501 13:58:29.078013 3935 tablet_bootstrap.cc:492] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:29.083526 3561 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38253
I20260501 13:58:29.084251 3867 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38253
I20260501 13:58:29.086163 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 36f21071de974647b03747d15e5f8ccc
I20260501 13:58:29.087963 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 36f21071de974647b03747d15e5f8ccc with delete_type TABLET_DATA_DELETED (replaced by tablet af227b4c28e34aa6abac901bdfb92415) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:29.089211 3227 catalog_manager.cc:6741] ReplaceTablet: tablet 36f21071de974647b03747d15e5f8ccc deleted and replaced by tablet af227b4c28e34aa6abac901bdfb92415
I20260501 13:58:29.091336 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 36f21071de974647b03747d15e5f8ccc with delete_type TABLET_DATA_DELETED (replaced by tablet af227b4c28e34aa6abac901bdfb92415) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:29.099695 3221 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 90fc78d9f46f409f91cf4193afe4398d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
W20260501 13:58:29.101342 3221 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 90fc78d9f46f409f91cf4193afe4398d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 90fc78d9f46f409f91cf4193afe4398d consensus state has no leader
I20260501 13:58:29.105862 3935 tablet_bootstrap.cc:654] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.107161 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet 36f21071de974647b03747d15e5f8ccc with delete_type TABLET_DATA_DELETED (replaced by tablet af227b4c28e34aa6abac901bdfb92415) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:29.109292 3936 tablet_bootstrap.cc:492] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:29.109807 3936 ts_tablet_manager.cc:1403] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.200s user 0.007s sys 0.025s
I20260501 13:58:29.111047 3942 ts_tablet_manager.cc:1929] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.111384 3942 log.cc:1199] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/7ac3e55605994b10899afcba0ba6abe9
I20260501 13:58:29.111943 3942 ts_tablet_manager.cc:1950] T 7ac3e55605994b10899afcba0ba6abe9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:29.115489 3936 raft_consensus.cc:359] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.116196 3936 raft_consensus.cc:385] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.116487 3936 raft_consensus.cc:740] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.116672 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 7ac3e55605994b10899afcba0ba6abe9 (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:29.117606 3942 tablet_replica.cc:333] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:29.118829 3942 raft_consensus.cc:2243] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.119583 3942 raft_consensus.cc:2272] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.121256 3221 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 90fc78d9f46f409f91cf4193afe4398d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:29.127960 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 7152aff22d5744d8919a82aa2e5f3798 (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.128980 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7152aff22d5744d8919a82aa2e5f3798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.131935 3942 ts_tablet_manager.cc:1916] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.132196 3719 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38253
I20260501 13:58:29.137718 3935 tablet_bootstrap.cc:492] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:29.138139 3935 ts_tablet_manager.cc:1403] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.000s
I20260501 13:58:29.139819 3719 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38253
I20260501 13:58:29.140548 3935 raft_consensus.cc:359] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.141139 3935 raft_consensus.cc:385] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.141366 3935 raft_consensus.cc:740] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.142074 3935 consensus_queue.cc:260] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.149488 4021 tablet_bootstrap.cc:492] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:29.149947 4021 ts_tablet_manager.cc:1403] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.096s user 0.010s sys 0.001s
I20260501 13:58:29.151429 3935 ts_tablet_manager.cc:1434] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20260501 13:58:29.153908 3801 tablet_service.cc:1511] Processing CreateTablet for tablet 7152aff22d5744d8919a82aa2e5f3798 (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.154924 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7152aff22d5744d8919a82aa2e5f3798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.160032 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 7ac3e55605994b10899afcba0ba6abe9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7152aff22d5744d8919a82aa2e5f3798) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:29.162807 3936 consensus_queue.cc:260] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.164717 4021 raft_consensus.cc:359] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.165359 4021 raft_consensus.cc:385] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.165611 4021 raft_consensus.cc:740] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.168227 3935 tablet_bootstrap.cc:492] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:29.168292 4021 consensus_queue.cc:260] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.175421 3936 ts_tablet_manager.cc:1434] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.065s user 0.005s sys 0.000s
I20260501 13:58:29.180877 3935 tablet_bootstrap.cc:654] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.181427 3943 ts_tablet_manager.cc:1916] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.187340 4021 ts_tablet_manager.cc:1434] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.037s user 0.001s sys 0.005s
I20260501 13:58:29.188062 3942 ts_tablet_manager.cc:1929] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.188422 3942 log.cc:1199] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/0d06ec6cc444404f9ec2eb3c91a48528
I20260501 13:58:29.189023 3942 ts_tablet_manager.cc:1950] T 0d06ec6cc444404f9ec2eb3c91a48528 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:29.191442 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 0d06ec6cc444404f9ec2eb3c91a48528 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
W20260501 13:58:29.200976 3213 catalog_manager.cc:4969] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet 8aa8986f92c4462fbc159f17cc282a0b because the tablet was not found. No further retry: Not found: Tablet not found: 8aa8986f92c4462fbc159f17cc282a0b
I20260501 13:58:29.202854 3935 tablet_bootstrap.cc:492] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:29.203266 3935 ts_tablet_manager.cc:1403] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.000s
I20260501 13:58:29.206727 3935 raft_consensus.cc:359] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.207345 3935 raft_consensus.cc:385] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.207579 3935 raft_consensus.cc:740] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.208124 3935 consensus_queue.cc:260] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:29.209966 3213 catalog_manager.cc:4982] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet 36f21071de974647b03747d15e5f8ccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36f21071de974647b03747d15e5f8ccc already in progress: creating tablet
I20260501 13:58:29.213280 3936 tablet_bootstrap.cc:492] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:29.213590 3221 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 90fc78d9f46f409f91cf4193afe4398d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:29.215322 3941 ts_tablet_manager.cc:1929] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.225284 4021 tablet_bootstrap.cc:492] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:29.217473 4116 log.cc:826] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Log is configured to *not* fsync() on all Append() calls
I20260501 13:58:29.230896 4021 tablet_bootstrap.cc:654] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.219434 3935 ts_tablet_manager.cc:1434] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260501 13:58:29.240401 3935 tablet_bootstrap.cc:492] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:29.242503 3941 log.cc:1199] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/0d06ec6cc444404f9ec2eb3c91a48528
I20260501 13:58:29.243424 3941 ts_tablet_manager.cc:1950] T 0d06ec6cc444404f9ec2eb3c91a48528 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:29.245571 3935 tablet_bootstrap.cc:654] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.260306 4021 tablet_bootstrap.cc:492] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:29.260728 4021 ts_tablet_manager.cc:1403] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.000s
I20260501 13:58:29.263198 4021 raft_consensus.cc:359] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.263844 4021 raft_consensus.cc:385] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.264104 4021 raft_consensus.cc:740] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.264688 3935 tablet_bootstrap.cc:492] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:29.265341 3935 ts_tablet_manager.cc:1403] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.000s
I20260501 13:58:29.246421 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 0d06ec6cc444404f9ec2eb3c91a48528 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:29.275069 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aa8986f92c4462fbc159f17cc282a0b with delete_type TABLET_DATA_DELETED (replaced by tablet 36f21071de974647b03747d15e5f8ccc) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:29.267916 3935 raft_consensus.cc:359] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.276139 3935 raft_consensus.cc:385] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.276408 3935 raft_consensus.cc:740] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.277227 3943 ts_tablet_manager.cc:1929] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:29.277093 3935 consensus_queue.cc:260] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:29.278636 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 36f21071de974647b03747d15e5f8ccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36f21071de974647b03747d15e5f8ccc already in progress: creating tablet
I20260501 13:58:29.280005 4021 consensus_queue.cc:260] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.282632 3935 ts_tablet_manager.cc:1434] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.017s user 0.006s sys 0.002s
I20260501 13:58:29.285148 3935 tablet_bootstrap.cc:492] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:29.290292 4021 ts_tablet_manager.cc:1434] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260501 13:58:29.277619 3943 log.cc:1199] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/0d06ec6cc444404f9ec2eb3c91a48528
I20260501 13:58:29.293814 3943 ts_tablet_manager.cc:1950] T 0d06ec6cc444404f9ec2eb3c91a48528 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:29.298530 3941 tablet_replica.cc:333] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:29.303431 3943 tablet_replica.cc:333] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:29.304149 3943 raft_consensus.cc:2243] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.304621 3943 raft_consensus.cc:2272] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.296936 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 0d06ec6cc444404f9ec2eb3c91a48528 (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:29.315953 3561 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38253
I20260501 13:58:29.318852 3941 raft_consensus.cc:2243] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.325517 3935 tablet_bootstrap.cc:654] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.325903 3941 raft_consensus.cc:2272] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.329874 3941 ts_tablet_manager.cc:1916] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.318432 3936 tablet_bootstrap.cc:654] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.331235 3943 ts_tablet_manager.cc:1916] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.352003 3227 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 90fc78d9f46f409f91cf4193afe4398d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:29.359093 3221 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 90fc78d9f46f409f91cf4193afe4398d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:29.360963 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet 8aa8986f92c4462fbc159f17cc282a0b from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:29.367487 3495 tablet_service.cc:1511] Processing CreateTablet for tablet af227b4c28e34aa6abac901bdfb92415 (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.368594 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af227b4c28e34aa6abac901bdfb92415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.375110 3947 raft_consensus.cc:493] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:29.375602 3947 raft_consensus.cc:515] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.377377 3947 leader_election.cc:290] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:29.380558 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b34fd361c8346938ce68b6e0e726d8d" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:29.381574 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b34fd361c8346938ce68b6e0e726d8d" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:29.382758 3673 raft_consensus.cc:2393] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9154033cea374a10a5225c53b8bcf122 in current term 1: Already voted for candidate 5c239ccbdc554619a237d78e958ffbc2 in this term.
I20260501 13:58:29.384349 3448 leader_election.cc:304] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:29.385118 3947 raft_consensus.cc:3060] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:29.386783 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 36f21071de974647b03747d15e5f8ccc with delete_type TABLET_DATA_DELETED (replaced by tablet af227b4c28e34aa6abac901bdfb92415) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:29.387835 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 36f21071de974647b03747d15e5f8ccc from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:29.392442 3652 tablet_service.cc:1511] Processing CreateTablet for tablet af227b4c28e34aa6abac901bdfb92415 (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.393446 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af227b4c28e34aa6abac901bdfb92415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.393297 3947 raft_consensus.cc:2749] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:29.404045 3936 tablet_bootstrap.cc:492] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:29.404515 3936 ts_tablet_manager.cc:1403] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.191s user 0.008s sys 0.003s
I20260501 13:58:29.407058 3936 raft_consensus.cc:359] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.407697 3936 raft_consensus.cc:385] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.408030 3936 raft_consensus.cc:740] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.410946 3801 tablet_service.cc:1511] Processing CreateTablet for tablet af227b4c28e34aa6abac901bdfb92415 (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.411970 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af227b4c28e34aa6abac901bdfb92415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.412303 3494 tablet_service.cc:1511] Processing CreateTablet for tablet be144bbf949844539848011a9963ac2d (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.413403 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be144bbf949844539848011a9963ac2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.423136 3935 tablet_bootstrap.cc:492] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:29.423564 3935 ts_tablet_manager.cc:1403] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.139s user 0.005s sys 0.012s
I20260501 13:58:29.425920 3935 raft_consensus.cc:359] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.426553 3935 raft_consensus.cc:385] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.426787 3935 raft_consensus.cc:740] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.425688 4116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260501 13:58:29.427366 3935 consensus_queue.cc:260] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.429167 3935 ts_tablet_manager.cc:1434] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260501 13:58:29.430049 3943 ts_tablet_manager.cc:1929] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.431682 3943 log.cc:1199] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/8aa8986f92c4462fbc159f17cc282a0b
I20260501 13:58:29.433961 3943 ts_tablet_manager.cc:1950] T 8aa8986f92c4462fbc159f17cc282a0b P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:29.433992 3935 tablet_bootstrap.cc:492] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:29.434811 4116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Bootstrap complete.
I20260501 13:58:29.442739 3802 tablet_service.cc:1511] Processing CreateTablet for tablet be144bbf949844539848011a9963ac2d (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.445003 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be144bbf949844539848011a9963ac2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.443920 3654 tablet_service.cc:1511] Processing CreateTablet for tablet be144bbf949844539848011a9963ac2d (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.450388 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be144bbf949844539848011a9963ac2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.450093 3936 consensus_queue.cc:260] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.452088 3936 ts_tablet_manager.cc:1434] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20260501 13:58:29.452844 3936 tablet_bootstrap.cc:492] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:29.460429 3941 ts_tablet_manager.cc:1929] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.460788 3941 log.cc:1199] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/7ac3e55605994b10899afcba0ba6abe9
I20260501 13:58:29.461402 3941 ts_tablet_manager.cc:1950] T 7ac3e55605994b10899afcba0ba6abe9 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:29.468679 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 8aa8986f92c4462fbc159f17cc282a0b (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:29.468865 3943 tablet_replica.cc:333] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:29.469391 3935 tablet_bootstrap.cc:654] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.469763 3943 raft_consensus.cc:2243] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.472074 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet 36f21071de974647b03747d15e5f8ccc with delete_type TABLET_DATA_DELETED (replaced by tablet af227b4c28e34aa6abac901bdfb92415) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:29.474735 3943 raft_consensus.cc:2272] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.476895 3942 tablet_replica.cc:333] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:29.477676 3942 raft_consensus.cc:2243] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.478124 3942 raft_consensus.cc:2272] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.480203 3942 ts_tablet_manager.cc:1916] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.482255 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 7ac3e55605994b10899afcba0ba6abe9 (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:29.483258 3941 tablet_replica.cc:333] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:29.484035 3941 raft_consensus.cc:2243] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.484464 3941 raft_consensus.cc:2272] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.495141 3943 ts_tablet_manager.cc:1916] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.503279 4021 tablet_bootstrap.cc:492] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:29.522558 3941 ts_tablet_manager.cc:1916] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.523025 3935 tablet_bootstrap.cc:492] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:29.523331 4021 tablet_bootstrap.cc:654] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.523407 3935 ts_tablet_manager.cc:1403] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.003s
I20260501 13:58:29.525923 3936 tablet_bootstrap.cc:654] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.525727 3935 raft_consensus.cc:359] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.528753 3942 ts_tablet_manager.cc:1929] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.529186 3942 log.cc:1199] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/36f21071de974647b03747d15e5f8ccc
I20260501 13:58:29.529757 3942 ts_tablet_manager.cc:1950] T 36f21071de974647b03747d15e5f8ccc P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:29.526330 3935 raft_consensus.cc:385] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.535070 3935 raft_consensus.cc:740] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.538013 3936 tablet_bootstrap.cc:492] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:29.538426 3936 ts_tablet_manager.cc:1403] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.007s
I20260501 13:58:29.540839 3936 raft_consensus.cc:359] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.541481 3936 raft_consensus.cc:385] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.541731 3936 raft_consensus.cc:740] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.542114 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 36f21071de974647b03747d15e5f8ccc (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:29.544440 4021 tablet_bootstrap.cc:492] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:29.544828 4021 ts_tablet_manager.cc:1403] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.002s
I20260501 13:58:29.547168 4021 raft_consensus.cc:359] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.547791 4021 raft_consensus.cc:385] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.542158 3935 consensus_queue.cc:260] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.548064 4021 raft_consensus.cc:740] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.548645 4021 consensus_queue.cc:260] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.549450 3941 ts_tablet_manager.cc:1929] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.550427 4021 ts_tablet_manager.cc:1434] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260501 13:58:29.551286 4021 tablet_bootstrap.cc:492] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:29.549804 3941 log.cc:1199] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/8aa8986f92c4462fbc159f17cc282a0b
I20260501 13:58:29.542364 3936 consensus_queue.cc:260] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.552254 3941 ts_tablet_manager.cc:1950] T 8aa8986f92c4462fbc159f17cc282a0b P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:29.554558 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 8aa8986f92c4462fbc159f17cc282a0b (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:29.563966 3936 ts_tablet_manager.cc:1434] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260501 13:58:29.577255 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 90fc78d9f46f409f91cf4193afe4398d from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:29.578449 3936 tablet_bootstrap.cc:492] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:29.588446 3935 ts_tablet_manager.cc:1434] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.065s user 0.007s sys 0.000s
I20260501 13:58:29.601919 4021 tablet_bootstrap.cc:654] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.601251 3935 tablet_bootstrap.cc:492] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:29.604004 3943 ts_tablet_manager.cc:1929] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.616959 3943 log.cc:1199] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/36f21071de974647b03747d15e5f8ccc
I20260501 13:58:29.617676 3943 ts_tablet_manager.cc:1950] T 36f21071de974647b03747d15e5f8ccc P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:29.619837 3936 tablet_bootstrap.cc:654] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.620071 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 36f21071de974647b03747d15e5f8ccc (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:29.621492 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 90fc78d9f46f409f91cf4193afe4398d
I20260501 13:58:29.623159 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 90fc78d9f46f409f91cf4193afe4398d with delete_type TABLET_DATA_DELETED (replaced by tablet 088e2909ee2548b78b56f8eb01f8edcb) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:29.627110 3941 tablet_replica.cc:333] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:29.630507 3941 raft_consensus.cc:2243] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.631268 4021 tablet_bootstrap.cc:492] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:29.631748 4021 ts_tablet_manager.cc:1403] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.002s
I20260501 13:58:29.632891 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 90fc78d9f46f409f91cf4193afe4398d with delete_type TABLET_DATA_DELETED (replaced by tablet 088e2909ee2548b78b56f8eb01f8edcb) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:29.633590 3942 tablet_replica.cc:333] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:29.634279 3942 raft_consensus.cc:2243] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.634753 3942 raft_consensus.cc:2272] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.636406 3942 ts_tablet_manager.cc:1916] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.631112 3941 raft_consensus.cc:2272] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.639192 3227 catalog_manager.cc:6741] ReplaceTablet: tablet 90fc78d9f46f409f91cf4193afe4398d deleted and replaced by tablet 088e2909ee2548b78b56f8eb01f8edcb
I20260501 13:58:29.639484 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 90fc78d9f46f409f91cf4193afe4398d with delete_type TABLET_DATA_DELETED (replaced by tablet 088e2909ee2548b78b56f8eb01f8edcb) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:29.640281 3943 tablet_replica.cc:333] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:29.640990 3943 raft_consensus.cc:2243] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.641492 3943 raft_consensus.cc:2272] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.642310 3941 ts_tablet_manager.cc:1916] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.643235 3943 ts_tablet_manager.cc:1916] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.647238 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 7152aff22d5744d8919a82aa2e5f3798 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:29.649262 3936 tablet_bootstrap.cc:492] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:29.650189 3936 ts_tablet_manager.cc:1403] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.072s user 0.007s sys 0.004s
I20260501 13:58:29.652592 3936 raft_consensus.cc:359] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.668401 3935 tablet_bootstrap.cc:654] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.638163 4021 raft_consensus.cc:359] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.669164 4021 raft_consensus.cc:385] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.669497 4021 raft_consensus.cc:740] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.670274 4021 consensus_queue.cc:260] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.671064 3936 raft_consensus.cc:385] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.671379 3936 raft_consensus.cc:740] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.672329 3936 consensus_queue.cc:260] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.680891 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 7152aff22d5744d8919a82aa2e5f3798
I20260501 13:58:29.682529 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 7152aff22d5744d8919a82aa2e5f3798 with delete_type TABLET_DATA_DELETED (replaced by tablet 02ffd79e183e46489b50e1c7aecee10f) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:29.684317 3936 ts_tablet_manager.cc:1434] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.034s user 0.004s sys 0.000s
I20260501 13:58:29.685106 3936 tablet_bootstrap.cc:492] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:29.686115 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 7152aff22d5744d8919a82aa2e5f3798 with delete_type TABLET_DATA_DELETED (replaced by tablet 02ffd79e183e46489b50e1c7aecee10f) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:29.689445 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 7152aff22d5744d8919a82aa2e5f3798 with delete_type TABLET_DATA_DELETED (replaced by tablet 02ffd79e183e46489b50e1c7aecee10f) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:29.690102 4021 ts_tablet_manager.cc:1434] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.057s user 0.006s sys 0.000s
I20260501 13:58:29.689921 3227 catalog_manager.cc:6741] ReplaceTablet: tablet 7152aff22d5744d8919a82aa2e5f3798 deleted and replaced by tablet 02ffd79e183e46489b50e1c7aecee10f
I20260501 13:58:29.694629 3867 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38253
I20260501 13:58:29.713438 3936 tablet_bootstrap.cc:654] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.718302 3947 raft_consensus.cc:493] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:29.718834 3947 raft_consensus.cc:515] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.719722 3943 ts_tablet_manager.cc:1929] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.720077 3943 log.cc:1199] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/90fc78d9f46f409f91cf4193afe4398d
I20260501 13:58:29.720715 3943 ts_tablet_manager.cc:1950] T 90fc78d9f46f409f91cf4193afe4398d P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:29.720770 3947 leader_election.cc:290] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:29.722934 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af227b4c28e34aa6abac901bdfb92415" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:29.723567 3822 raft_consensus.cc:2468] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:29.725085 3448 leader_election.cc:304] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:29.722585 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af227b4c28e34aa6abac901bdfb92415" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:29.726833 3673 raft_consensus.cc:2468] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:29.727035 3947 raft_consensus.cc:2804] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:29.727489 3947 raft_consensus.cc:493] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:29.728464 3947 raft_consensus.cc:3060] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:29.730536 3943 tablet_replica.cc:333] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:29.730628 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 90fc78d9f46f409f91cf4193afe4398d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:29.731284 3943 raft_consensus.cc:2243] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.731843 3943 raft_consensus.cc:2272] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.734551 3942 ts_tablet_manager.cc:1929] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.734946 3942 log.cc:1199] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/90fc78d9f46f409f91cf4193afe4398d
I20260501 13:58:29.735584 3942 ts_tablet_manager.cc:1950] T 90fc78d9f46f409f91cf4193afe4398d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:29.739518 4027 consensus_queue.cc:1048] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:29.743045 3494 tablet_service.cc:1511] Processing CreateTablet for tablet 088e2909ee2548b78b56f8eb01f8edcb (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.744112 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 088e2909ee2548b78b56f8eb01f8edcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.756333 3943 ts_tablet_manager.cc:1916] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.759469 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 088e2909ee2548b78b56f8eb01f8edcb (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.760730 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 088e2909ee2548b78b56f8eb01f8edcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.772900 3947 raft_consensus.cc:515] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.776058 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af227b4c28e34aa6abac901bdfb92415" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:29.776736 3674 raft_consensus.cc:3060] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:29.779476 4027 raft_consensus.cc:493] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:29.779961 4027 raft_consensus.cc:515] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.781808 3935 tablet_bootstrap.cc:492] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:29.781965 4027 leader_election.cc:290] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:29.782393 3935 ts_tablet_manager.cc:1403] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.182s user 0.014s sys 0.000s
I20260501 13:58:29.783212 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 088e2909ee2548b78b56f8eb01f8edcb (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.785926 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "be144bbf949844539848011a9963ac2d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:29.786849 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "be144bbf949844539848011a9963ac2d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:29.785394 3935 raft_consensus.cc:359] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.787881 3935 raft_consensus.cc:385] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.788400 3935 raft_consensus.cc:740] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.739349 4116 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
W20260501 13:58:29.788741 3756 leader_election.cc:343] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:29.789263 4116 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0f8911dd5d431d9d8d3124a4456262, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.790709 4116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:29.794454 3674 raft_consensus.cc:2468] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:29.794656 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 90fc78d9f46f409f91cf4193afe4398d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:29.796001 3448 leader_election.cc:304] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:29.798966 3947 leader_election.cc:290] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:29.795883 3942 tablet_replica.cc:333] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:29.799503 3947 raft_consensus.cc:2804] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:29.799937 3947 raft_consensus.cc:697] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:29.800753 3942 raft_consensus.cc:2243] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.800863 3947 consensus_queue.cc:237] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.789330 3935 consensus_queue.cc:260] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.807039 3515 raft_consensus.cc:2468] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:29.811077 3936 tablet_bootstrap.cc:492] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:29.811499 3936 ts_tablet_manager.cc:1403] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.127s user 0.014s sys 0.025s
I20260501 13:58:29.812054 3757 leader_election.cc:304] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:29.812566 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af227b4c28e34aa6abac901bdfb92415" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:29.813123 3822 raft_consensus.cc:3060] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:29.813520 4027 raft_consensus.cc:2804] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:29.813858 4027 raft_consensus.cc:493] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:29.814196 4027 raft_consensus.cc:3060] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:29.813902 3936 raft_consensus.cc:359] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.814522 3936 raft_consensus.cc:385] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.814761 3936 raft_consensus.cc:740] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.815415 3936 consensus_queue.cc:260] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.817190 3936 ts_tablet_manager.cc:1434] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260501 13:58:29.818022 3936 tablet_bootstrap.cc:492] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:29.818027 3942 raft_consensus.cc:2272] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.825477 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 088e2909ee2548b78b56f8eb01f8edcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.826538 3822 raft_consensus.cc:2468] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:29.823871 3941 ts_tablet_manager.cc:1929] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.830699 3941 log.cc:1199] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/36f21071de974647b03747d15e5f8ccc
I20260501 13:58:29.836997 3935 ts_tablet_manager.cc:1434] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20260501 13:58:29.836982 3719 heartbeater.cc:344] Connected to a master server at 127.1.16.188:38253
I20260501 13:58:29.838362 3942 ts_tablet_manager.cc:1916] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.841836 4027 raft_consensus.cc:515] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.849460 4027 leader_election.cc:290] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:29.847941 4145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:29.851047 4145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:29.855283 3935 tablet_bootstrap.cc:492] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:29.856240 4116 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 13:58:29.864032 4147 consensus_queue.cc:1048] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:58:29.864470 4080 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 4 FOLLOWER]: Advancing to term 5
I20260501 13:58:29.837668 3941 ts_tablet_manager.cc:1950] T 36f21071de974647b03747d15e5f8ccc P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:29.852752 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "be144bbf949844539848011a9963ac2d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:29.867226 3515 raft_consensus.cc:3060] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:29.867700 3941 tablet_replica.cc:333] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:29.868397 3941 raft_consensus.cc:2243] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:29.868832 3941 raft_consensus.cc:2272] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:29.871887 3936 tablet_bootstrap.cc:654] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.873728 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 36f21071de974647b03747d15e5f8ccc (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:29.869422 3943 ts_tablet_manager.cc:1929] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.883890 3943 log.cc:1199] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/7152aff22d5744d8919a82aa2e5f3798
I20260501 13:58:29.884567 3943 ts_tablet_manager.cc:1950] T 7152aff22d5744d8919a82aa2e5f3798 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
W20260501 13:58:29.872314 3756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.130:45857, user_credentials={real_user=slave}} blocked reactor thread for 53213.6us
I20260501 13:58:29.886875 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 7152aff22d5744d8919a82aa2e5f3798 (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:29.888005 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "be144bbf949844539848011a9963ac2d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:29.888579 3673 raft_consensus.cc:3060] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:29.881743 4021 tablet_bootstrap.cc:492] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:29.892951 3515 raft_consensus.cc:2468] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:29.894543 3941 ts_tablet_manager.cc:1916] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:29.895892 3673 raft_consensus.cc:2468] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:29.900800 3494 tablet_service.cc:1511] Processing CreateTablet for tablet 02ffd79e183e46489b50e1c7aecee10f (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.902045 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ffd79e183e46489b50e1c7aecee10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.905669 3756 leader_election.cc:304] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:29.917222 4021 tablet_bootstrap.cc:654] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.919178 3936 tablet_bootstrap.cc:492] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:29.919571 3936 ts_tablet_manager.cc:1403] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.102s user 0.009s sys 0.004s
I20260501 13:58:29.923698 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 02ffd79e183e46489b50e1c7aecee10f (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.924796 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ffd79e183e46489b50e1c7aecee10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.932176 3942 ts_tablet_manager.cc:1929] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.932193 4159 raft_consensus.cc:2804] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:29.932588 3942 log.cc:1199] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/7152aff22d5744d8919a82aa2e5f3798
I20260501 13:58:29.933254 3942 ts_tablet_manager.cc:1950] T 7152aff22d5744d8919a82aa2e5f3798 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:29.940687 3936 raft_consensus.cc:359] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.941366 3936 raft_consensus.cc:385] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.941617 3936 raft_consensus.cc:740] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.945274 4021 tablet_bootstrap.cc:492] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:29.945851 4021 ts_tablet_manager.cc:1403] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.000s
I20260501 13:58:29.948596 4080 pending_rounds.cc:85] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Aborting all ops after (but not including) 5
I20260501 13:58:29.948891 4080 pending_rounds.cc:107] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260501 13:58:29.949196 4080 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 13:58:29.949190 4021 raft_consensus.cc:359] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.949787 4021 raft_consensus.cc:385] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:29.950026 4021 raft_consensus.cc:740] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:29.950731 4021 consensus_queue.cc:260] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.952965 4021 ts_tablet_manager.cc:1434] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260501 13:58:29.954914 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 7152aff22d5744d8919a82aa2e5f3798 (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:29.962159 3935 tablet_bootstrap.cc:654] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:29.942247 3936 consensus_queue.cc:260] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.967943 4008 raft_consensus.cc:493] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:29.968390 4008 raft_consensus.cc:515] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.970867 4008 leader_election.cc:290] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:29.972143 4159 raft_consensus.cc:697] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:29.973028 4159 consensus_queue.cc:237] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:29.975039 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 02ffd79e183e46489b50e1c7aecee10f (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:29.976073 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ffd79e183e46489b50e1c7aecee10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:29.976666 3936 ts_tablet_manager.cc:1434] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.057s user 0.005s sys 0.002s
I20260501 13:58:29.978729 3514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "088e2909ee2548b78b56f8eb01f8edcb" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
W20260501 13:58:29.980388 3609 leader_election.cc:343] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:29.993237 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "088e2909ee2548b78b56f8eb01f8edcb" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:29.993862 3822 raft_consensus.cc:2468] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:29.995476 3608 leader_election.cc:304] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:29.996490 3941 ts_tablet_manager.cc:1929] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:29.996572 4008 raft_consensus.cc:2804] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:29.996847 3941 log.cc:1199] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/90fc78d9f46f409f91cf4193afe4398d
I20260501 13:58:29.996887 4008 raft_consensus.cc:493] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:29.997179 4008 raft_consensus.cc:3060] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:29.997470 3941 ts_tablet_manager.cc:1950] T 90fc78d9f46f409f91cf4193afe4398d P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:30.001653 4021 tablet_bootstrap.cc:492] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:30.009846 4021 tablet_bootstrap.cc:654] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:30.012583 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 90fc78d9f46f409f91cf4193afe4398d from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:30.019857 3221 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 088e2909ee2548b78b56f8eb01f8edcb (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
W20260501 13:58:30.021759 3221 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 088e2909ee2548b78b56f8eb01f8edcb (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 088e2909ee2548b78b56f8eb01f8edcb consensus state has no leader
I20260501 13:58:30.024808 4021 tablet_bootstrap.cc:492] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:30.025213 4021 ts_tablet_manager.cc:1403] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.004s
I20260501 13:58:30.027762 4021 raft_consensus.cc:359] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.028394 4021 raft_consensus.cc:385] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:30.028074 4008 raft_consensus.cc:515] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.028674 4021 raft_consensus.cc:740] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:30.033352 4021 consensus_queue.cc:260] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.035522 4021 ts_tablet_manager.cc:1434] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260501 13:58:30.037549 3221 catalog_manager.cc:5671] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } }
I20260501 13:58:30.015799 3227 catalog_manager.cc:5671] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9154033cea374a10a5225c53b8bcf122 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "9154033cea374a10a5225c53b8bcf122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
W20260501 13:58:30.054104 3214 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 088e2909ee2548b78b56f8eb01f8edcb (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 088e2909ee2548b78b56f8eb01f8edcb consensus state has no leader
I20260501 13:58:30.062887 3514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "088e2909ee2548b78b56f8eb01f8edcb" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
W20260501 13:58:30.064242 3609 leader_election.cc:343] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:30.065331 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 90fc78d9f46f409f91cf4193afe4398d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:30.067502 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "088e2909ee2548b78b56f8eb01f8edcb" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:30.068069 3822 raft_consensus.cc:3060] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:30.071615 3941 tablet_replica.cc:333] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:30.072324 3941 raft_consensus.cc:2243] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:30.072786 3941 raft_consensus.cc:2272] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:30.075523 3822 raft_consensus.cc:2468] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:30.076757 3608 leader_election.cc:304] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:30.077878 4163 raft_consensus.cc:2804] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:30.082439 4163 raft_consensus.cc:697] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:30.083448 4163 consensus_queue.cc:237] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.086208 3941 ts_tablet_manager.cc:1916] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:30.093933 4008 leader_election.cc:290] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:30.081501 3227 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 088e2909ee2548b78b56f8eb01f8edcb (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:30.153980 3941 ts_tablet_manager.cc:1929] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:30.154351 3941 log.cc:1199] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/7152aff22d5744d8919a82aa2e5f3798
I20260501 13:58:30.155045 3941 ts_tablet_manager.cc:1950] T 7152aff22d5744d8919a82aa2e5f3798 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:30.157281 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 7152aff22d5744d8919a82aa2e5f3798 (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:30.193589 3227 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 088e2909ee2548b78b56f8eb01f8edcb (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:30.208154 3935 tablet_bootstrap.cc:492] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:30.208676 3935 ts_tablet_manager.cc:1403] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.354s user 0.010s sys 0.000s
I20260501 13:58:30.211275 3935 raft_consensus.cc:359] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.214025 3936 tablet_bootstrap.cc:492] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:30.214363 3935 raft_consensus.cc:385] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:30.214730 3935 raft_consensus.cc:740] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:30.219751 3936 tablet_bootstrap.cc:654] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:30.225386 3935 consensus_queue.cc:260] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.231499 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 088e2909ee2548b78b56f8eb01f8edcb from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:30.243110 3936 tablet_bootstrap.cc:492] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:30.244475 3936 ts_tablet_manager.cc:1403] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.002s
I20260501 13:58:30.245713 3935 ts_tablet_manager.cc:1434] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20260501 13:58:30.246870 3936 raft_consensus.cc:359] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.247494 3936 raft_consensus.cc:385] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:30.247771 3936 raft_consensus.cc:740] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:30.248493 3936 consensus_queue.cc:260] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.270251 3935 tablet_bootstrap.cc:492] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:30.282987 3935 tablet_bootstrap.cc:654] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:30.250859 3936 ts_tablet_manager.cc:1434] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260501 13:58:30.326922 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 088e2909ee2548b78b56f8eb01f8edcb with delete_type TABLET_DATA_DELETED (replaced by tablet 034a29eda5fe4b43ac6dcf3616299e5d) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:30.327656 3941 tablet_replica.cc:333] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:30.328352 3941 raft_consensus.cc:2243] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:30.328773 3941 raft_consensus.cc:2272] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:30.319073 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 088e2909ee2548b78b56f8eb01f8edcb
I20260501 13:58:30.331034 3941 ts_tablet_manager.cc:1916] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:30.331030 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 088e2909ee2548b78b56f8eb01f8edcb with delete_type TABLET_DATA_DELETED (replaced by tablet 034a29eda5fe4b43ac6dcf3616299e5d) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:30.333269 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 088e2909ee2548b78b56f8eb01f8edcb with delete_type TABLET_DATA_DELETED (replaced by tablet 034a29eda5fe4b43ac6dcf3616299e5d) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:30.334190 3942 tablet_replica.cc:333] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:30.334443 3227 catalog_manager.cc:6741] ReplaceTablet: tablet 088e2909ee2548b78b56f8eb01f8edcb deleted and replaced by tablet 034a29eda5fe4b43ac6dcf3616299e5d
I20260501 13:58:30.334981 3942 raft_consensus.cc:2243] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:30.335275 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 7b34fd361c8346938ce68b6e0e726d8d from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:30.335611 3942 pending_rounds.cc:70] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: Trying to abort 1 pending ops.
I20260501 13:58:30.335814 3942 pending_rounds.cc:77] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229455712874496 op_type: NO_OP noop_request { }
I20260501 13:58:30.336164 3942 raft_consensus.cc:2889] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:30.336453 3942 raft_consensus.cc:2272] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:30.339136 3942 ts_tablet_manager.cc:1916] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:30.348017 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 088e2909ee2548b78b56f8eb01f8edcb with delete_type TABLET_DATA_DELETED (replaced by tablet 034a29eda5fe4b43ac6dcf3616299e5d) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:30.348428 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 088e2909ee2548b78b56f8eb01f8edcb with delete_type TABLET_DATA_DELETED (replaced by tablet 034a29eda5fe4b43ac6dcf3616299e5d) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:30.348726 3943 tablet_replica.cc:333] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:30.349411 3943 raft_consensus.cc:2243] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:30.349880 3943 raft_consensus.cc:2272] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:30.352170 3943 ts_tablet_manager.cc:1916] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:30.381202 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 7b34fd361c8346938ce68b6e0e726d8d
I20260501 13:58:30.383028 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 7b34fd361c8346938ce68b6e0e726d8d with delete_type TABLET_DATA_DELETED (replaced by tablet 7654c30b114a40aca4c79ca050e2c877) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:30.385197 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 7b34fd361c8346938ce68b6e0e726d8d with delete_type TABLET_DATA_DELETED (replaced by tablet 7654c30b114a40aca4c79ca050e2c877) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:30.386276 3941 ts_tablet_manager.cc:1929] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:30.386700 3941 log.cc:1199] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/088e2909ee2548b78b56f8eb01f8edcb
I20260501 13:58:30.387356 3941 ts_tablet_manager.cc:1950] T 088e2909ee2548b78b56f8eb01f8edcb P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:30.388060 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 7b34fd361c8346938ce68b6e0e726d8d with delete_type TABLET_DATA_DELETED (replaced by tablet 7654c30b114a40aca4c79ca050e2c877) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:30.390007 3941 tablet_replica.cc:333] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:30.391997 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 088e2909ee2548b78b56f8eb01f8edcb (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
W20260501 13:58:30.392832 3214 catalog_manager.cc:4969] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 088e2909ee2548b78b56f8eb01f8edcb because the tablet was not found. No further retry: Not found: Tablet not found: 088e2909ee2548b78b56f8eb01f8edcb
I20260501 13:58:30.393271 3942 ts_tablet_manager.cc:1929] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:30.393719 3942 log.cc:1199] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/088e2909ee2548b78b56f8eb01f8edcb
I20260501 13:58:30.392307 3943 ts_tablet_manager.cc:1929] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:30.394791 3942 ts_tablet_manager.cc:1950] T 088e2909ee2548b78b56f8eb01f8edcb P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:30.397490 3942 tablet_replica.cc:333] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:30.397861 3943 log.cc:1199] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/088e2909ee2548b78b56f8eb01f8edcb
I20260501 13:58:30.398578 3943 ts_tablet_manager.cc:1950] T 088e2909ee2548b78b56f8eb01f8edcb P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:30.400175 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 088e2909ee2548b78b56f8eb01f8edcb (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:30.400985 3943 tablet_replica.cc:333] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
W20260501 13:58:30.400938 3213 catalog_manager.cc:4969] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet 088e2909ee2548b78b56f8eb01f8edcb because the tablet was not found. No further retry: Not found: Tablet not found: 088e2909ee2548b78b56f8eb01f8edcb
I20260501 13:58:30.401546 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 088e2909ee2548b78b56f8eb01f8edcb (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:30.403160 3942 raft_consensus.cc:2243] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:30.404186 3943 raft_consensus.cc:2243] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:30.405241 3941 raft_consensus.cc:2243] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:30.407522 3227 catalog_manager.cc:6741] ReplaceTablet: tablet 7b34fd361c8346938ce68b6e0e726d8d deleted and replaced by tablet 7654c30b114a40aca4c79ca050e2c877
I20260501 13:58:30.408735 3943 raft_consensus.cc:2272] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:30.409354 3942 raft_consensus.cc:2272] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:30.410012 3941 raft_consensus.cc:2272] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:30.410671 4163 raft_consensus.cc:493] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:30.411163 4163 raft_consensus.cc:515] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.413094 4163 leader_election.cc:290] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:30.414757 3514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02ffd79e183e46489b50e1c7aecee10f" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:30.415627 3941 ts_tablet_manager.cc:1916] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:58:30.416766 3609 leader_election.cc:343] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:30.416610 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02ffd79e183e46489b50e1c7aecee10f" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:30.418848 3822 raft_consensus.cc:2468] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:30.421435 3608 leader_election.cc:304] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:30.422106 3943 ts_tablet_manager.cc:1916] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:30.422557 4163 raft_consensus.cc:2804] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:30.422911 4163 raft_consensus.cc:493] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:30.423235 4163 raft_consensus.cc:3060] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:30.429404 3942 ts_tablet_manager.cc:1916] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:30.429837 3494 tablet_service.cc:1511] Processing CreateTablet for tablet 034a29eda5fe4b43ac6dcf3616299e5d (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:30.430912 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 034a29eda5fe4b43ac6dcf3616299e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:30.434494 4163 raft_consensus.cc:515] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.438445 4145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15dd70d7c5494c3c911f0a780dc9a2af. Latest consensus state: current_term: 5 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:30.443457 4145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:30.444713 4179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:30.446307 4179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:30.446517 3514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02ffd79e183e46489b50e1c7aecee10f" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:30.446606 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02ffd79e183e46489b50e1c7aecee10f" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:30.447111 3822 raft_consensus.cc:3060] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:58:30.447906 3609 leader_election.cc:343] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:30.453938 3822 raft_consensus.cc:2468] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:30.455150 3608 leader_election.cc:304] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:30.455953 4008 raft_consensus.cc:2804] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:30.456513 4008 raft_consensus.cc:697] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:30.457289 4008 consensus_queue.cc:237] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.466959 4080 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35420 (request call id 0) took 1398 ms. Trace:
I20260501 13:58:30.471901 4163 leader_election.cc:290] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:30.480067 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 034a29eda5fe4b43ac6dcf3616299e5d (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:30.481123 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 034a29eda5fe4b43ac6dcf3616299e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:30.488643 4188 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260501 13:58:30.500168 3941 ts_tablet_manager.cc:1929] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:30.500571 3941 log.cc:1199] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/7b34fd361c8346938ce68b6e0e726d8d
I20260501 13:58:30.501256 3941 ts_tablet_manager.cc:1950] T 7b34fd361c8346938ce68b6e0e726d8d P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:30.501158 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 034a29eda5fe4b43ac6dcf3616299e5d (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:30.502207 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 034a29eda5fe4b43ac6dcf3616299e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:30.504094 3935 tablet_bootstrap.cc:492] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:30.504494 3935 ts_tablet_manager.cc:1403] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.234s user 0.020s sys 0.024s
I20260501 13:58:30.506863 3935 raft_consensus.cc:359] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.507472 3935 raft_consensus.cc:385] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:30.507718 3935 raft_consensus.cc:740] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:30.508296 3935 consensus_queue.cc:260] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.511070 3942 ts_tablet_manager.cc:1929] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:30.518570 3943 ts_tablet_manager.cc:1929] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:30.519093 3942 log.cc:1199] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/7b34fd361c8346938ce68b6e0e726d8d
I20260501 13:58:30.519500 3943 log.cc:1199] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/7b34fd361c8346938ce68b6e0e726d8d
I20260501 13:58:30.522688 3227 catalog_manager.cc:5671] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:30.524883 3942 ts_tablet_manager.cc:1950] T 7b34fd361c8346938ce68b6e0e726d8d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:30.526248 3935 ts_tablet_manager.cc:1434] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20260501 13:58:30.527135 3943 ts_tablet_manager.cc:1950] T 7b34fd361c8346938ce68b6e0e726d8d P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:30.534652 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 7b34fd361c8346938ce68b6e0e726d8d (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:30.536885 4188 catalog_manager.cc:1269] Loaded cluster ID: 4281e294e62c41c781e2041e1b24de7c
I20260501 13:58:30.544034 4021 tablet_bootstrap.cc:492] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:30.545521 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 7b34fd361c8346938ce68b6e0e726d8d (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:30.546228 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 7b34fd361c8346938ce68b6e0e726d8d (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:30.552254 4021 tablet_bootstrap.cc:654] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:30.554111 4188 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: loading cluster ID for follower catalog manager: success
I20260501 13:58:30.564632 4061 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9154033cea374a10a5225c53b8bcf122" instance_seqno: 1777643898269658) as {username='slave'} at 127.1.16.129:45709; Asking this server to re-register.
I20260501 13:58:30.567152 3561 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:30.572335 4066 ts_manager.cc:194] Registered new tserver with Master: 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:30.576407 4061 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" instance_seqno: 1777643900635713) as {username='slave'} at 127.1.16.130:40453; Asking this server to re-register.
I20260501 13:58:30.579123 3719 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:30.586633 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 088e2909ee2548b78b56f8eb01f8edcb with delete_type TABLET_DATA_DELETED (replaced by tablet 034a29eda5fe4b43ac6dcf3616299e5d) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:30.589277 4188 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: acquiring CA information for follower catalog manager: success
I20260501 13:58:30.590912 4021 tablet_bootstrap.cc:492] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
W20260501 13:58:30.590845 3214 catalog_manager.cc:4969] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 088e2909ee2548b78b56f8eb01f8edcb because the tablet was not found. No further retry: Not found: Tablet not found: 088e2909ee2548b78b56f8eb01f8edcb
I20260501 13:58:30.591315 4021 ts_tablet_manager.cc:1403] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.019s
I20260501 13:58:30.591753 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet 02ffd79e183e46489b50e1c7aecee10f from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:30.593658 4021 raft_consensus.cc:359] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.594260 4021 raft_consensus.cc:385] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:30.594436 4061 ts_manager.cc:194] Registered new tserver with Master: 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:30.598875 3935 tablet_bootstrap.cc:492] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:30.601661 4188 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 13:58:30.604265 3935 tablet_bootstrap.cc:654] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:30.609526 4021 raft_consensus.cc:740] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:30.610245 4021 consensus_queue.cc:260] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.615298 4179 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 15dd70d7c5494c3c911f0a780dc9a2af)
I20260501 13:58:30.615849 4179 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:30.620591 4179 leader_election.cc:290] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255)
I20260501 13:58:30.643788 3935 tablet_bootstrap.cc:492] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:30.644224 3935 ts_tablet_manager.cc:1403] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.000s
I20260501 13:58:30.644398 4061 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" instance_seqno: 1777643902876599) as {username='slave'} at 127.1.16.131:43381; Asking this server to re-register.
I20260501 13:58:30.646622 3867 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:30.647764 3935 raft_consensus.cc:359] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.648558 3935 raft_consensus.cc:385] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:30.648809 3935 raft_consensus.cc:740] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:30.649402 3935 consensus_queue.cc:260] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.651278 3935 ts_tablet_manager.cc:1434] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260501 13:58:30.654369 3947 raft_consensus.cc:493] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:30.654896 3947 raft_consensus.cc:515] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.655953 4061 ts_manager.cc:194] Registered new tserver with Master: 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:30.656754 3936 tablet_bootstrap.cc:492] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:30.656785 3947 leader_election.cc:290] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:30.662288 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02ffd79e183e46489b50e1c7aecee10f" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:30.662861 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02ffd79e183e46489b50e1c7aecee10f" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:30.663717 3822 raft_consensus.cc:2393] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9154033cea374a10a5225c53b8bcf122 in current term 1: Already voted for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in this term.
I20260501 13:58:30.666640 3936 tablet_bootstrap.cc:654] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:30.667799 4021 ts_tablet_manager.cc:1434] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.076s user 0.002s sys 0.012s
I20260501 13:58:30.671766 3448 leader_election.cc:304] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:30.673120 3947 raft_consensus.cc:3060] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:30.469358 4080 rpcz_store.cc:276] 0501 13:58:29.068728 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:29.068916 (+ 188us) service_pool.cc:224] Handling call
0501 13:58:29.859570 (+790654us) raft_consensus.cc:1471] Updating replica for 63 ops
0501 13:58:30.114341 (+254771us) raft_consensus.cc:1534] Early marking committed up to index 0
0501 13:58:30.114382 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 63 ops
0501 13:58:30.126697 (+ 12315us) write_op.cc:270] Start()
0501 13:58:30.126824 (+ 127us) write_op.cc:276] Timestamp: P: 1777643903924609 usec, L: 0
0501 13:58:30.158583 (+ 31759us) write_op.cc:270] Start()
0501 13:58:30.158687 (+ 104us) write_op.cc:276] Timestamp: P: 1777643904129109 usec, L: 0
0501 13:58:30.159371 (+ 684us) write_op.cc:270] Start()
0501 13:58:30.159491 (+ 120us) write_op.cc:276] Timestamp: P: 1777643904159819 usec, L: 0
0501 13:58:30.160016 (+ 525us) write_op.cc:270] Start()
0501 13:58:30.160109 (+ 93us) write_op.cc:276] Timestamp: P: 1777643904336485 usec, L: 0
0501 13:58:30.160579 (+ 470us) write_op.cc:270] Start()
0501 13:58:30.160655 (+ 76us) write_op.cc:276] Timestamp: P: 1777643904382592 usec, L: 0
0501 13:58:30.161095 (+ 440us) write_op.cc:270] Start()
0501 13:58:30.179200 (+ 18105us) write_op.cc:276] Timestamp: P: 1777643904588190 usec, L: 0
0501 13:58:30.179860 (+ 660us) write_op.cc:270] Start()
0501 13:58:30.180030 (+ 170us) write_op.cc:276] Timestamp: P: 1777643904640042 usec, L: 0
0501 13:58:30.180559 (+ 529us) write_op.cc:270] Start()
0501 13:58:30.180641 (+ 82us) write_op.cc:276] Timestamp: P: 1777643904846133 usec, L: 0
0501 13:58:30.181165 (+ 524us) write_op.cc:270] Start()
0501 13:58:30.181246 (+ 81us) write_op.cc:276] Timestamp: P: 1777643904874271 usec, L: 0
0501 13:58:30.181720 (+ 474us) write_op.cc:270] Start()
0501 13:58:30.181796 (+ 76us) write_op.cc:276] Timestamp: P: 1777643904879194 usec, L: 0
0501 13:58:30.182279 (+ 483us) write_op.cc:270] Start()
0501 13:58:30.182372 (+ 93us) write_op.cc:276] Timestamp: P: 1777643905029133 usec, L: 0
0501 13:58:30.223899 (+ 41527us) write_op.cc:270] Start()
0501 13:58:30.224027 (+ 128us) write_op.cc:276] Timestamp: P: 1777643905041320 usec, L: 0
0501 13:58:30.224627 (+ 600us) write_op.cc:270] Start()
0501 13:58:30.224707 (+ 80us) write_op.cc:276] Timestamp: P: 1777643905046361 usec, L: 0
0501 13:58:30.225180 (+ 473us) write_op.cc:270] Start()
0501 13:58:30.225255 (+ 75us) write_op.cc:276] Timestamp: P: 1777643905140693 usec, L: 0
0501 13:58:30.225707 (+ 452us) write_op.cc:270] Start()
0501 13:58:30.225778 (+ 71us) write_op.cc:276] Timestamp: P: 1777643905168695 usec, L: 0
0501 13:58:30.226226 (+ 448us) write_op.cc:270] Start()
0501 13:58:30.226304 (+ 78us) write_op.cc:276] Timestamp: P: 1777643905267680 usec, L: 0
0501 13:58:30.238528 (+ 12224us) write_op.cc:270] Start()
0501 13:58:30.238631 (+ 103us) write_op.cc:276] Timestamp: P: 1777643905439489 usec, L: 0
0501 13:58:30.239185 (+ 554us) write_op.cc:270] Start()
0501 13:58:30.239277 (+ 92us) write_op.cc:276] Timestamp: P: 1777643905469911 usec, L: 0
0501 13:58:30.239764 (+ 487us) write_op.cc:270] Start()
0501 13:58:30.239844 (+ 80us) write_op.cc:276] Timestamp: P: 1777643905551160 usec, L: 0
0501 13:58:30.262976 (+ 23132us) write_op.cc:270] Start()
0501 13:58:30.263088 (+ 112us) write_op.cc:276] Timestamp: P: 1777643905746425 usec, L: 0
0501 13:58:30.263640 (+ 552us) write_op.cc:270] Start()
0501 13:58:30.263722 (+ 82us) write_op.cc:276] Timestamp: P: 1777643905819942 usec, L: 0
0501 13:58:30.264209 (+ 487us) write_op.cc:270] Start()
0501 13:58:30.264294 (+ 85us) write_op.cc:276] Timestamp: P: 1777643906024906 usec, L: 0
0501 13:58:30.264766 (+ 472us) write_op.cc:270] Start()
0501 13:58:30.264844 (+ 78us) write_op.cc:276] Timestamp: P: 1777643906115125 usec, L: 0
0501 13:58:30.291299 (+ 26455us) write_op.cc:270] Start()
0501 13:58:30.291445 (+ 146us) write_op.cc:276] Timestamp: P: 1777643906195317 usec, L: 0
0501 13:58:30.292128 (+ 683us) write_op.cc:270] Start()
0501 13:58:30.292219 (+ 91us) write_op.cc:276] Timestamp: P: 1777643906266213 usec, L: 0
0501 13:58:30.292705 (+ 486us) write_op.cc:270] Start()
0501 13:58:30.292780 (+ 75us) write_op.cc:276] Timestamp: P: 1777643906270785 usec, L: 0
0501 13:58:30.293264 (+ 484us) write_op.cc:270] Start()
0501 13:58:30.293347 (+ 83us) write_op.cc:276] Timestamp: P: 1777643906349095 usec, L: 0
0501 13:58:30.293824 (+ 477us) write_op.cc:270] Start()
0501 13:58:30.293905 (+ 81us) write_op.cc:276] Timestamp: P: 1777643906505133 usec, L: 0
0501 13:58:30.294387 (+ 482us) write_op.cc:270] Start()
0501 13:58:30.310406 (+ 16019us) write_op.cc:276] Timestamp: P: 1777643906538145 usec, L: 0
0501 13:58:30.311108 (+ 702us) write_op.cc:270] Start()
0501 13:58:30.311202 (+ 94us) write_op.cc:276] Timestamp: P: 1777643906737770 usec, L: 0
0501 13:58:30.311705 (+ 503us) write_op.cc:270] Start()
0501 13:58:30.311783 (+ 78us) write_op.cc:276] Timestamp: P: 1777643906741205 usec, L: 0
0501 13:58:30.312243 (+ 460us) write_op.cc:270] Start()
0501 13:58:30.312324 (+ 81us) write_op.cc:276] Timestamp: P: 1777643906802210 usec, L: 0
0501 13:58:30.312879 (+ 555us) write_op.cc:270] Start()
0501 13:58:30.312960 (+ 81us) write_op.cc:276] Timestamp: P: 1777643906851944 usec, L: 0
0501 13:58:30.313415 (+ 455us) write_op.cc:270] Start()
0501 13:58:30.313496 (+ 81us) write_op.cc:276] Timestamp: P: 1777643907017107 usec, L: 0
0501 13:58:30.313960 (+ 464us) write_op.cc:270] Start()
0501 13:58:30.314378 (+ 418us) write_op.cc:276] Timestamp: P: 1777643907045577 usec, L: 0
0501 13:58:30.314905 (+ 527us) write_op.cc:270] Start()
0501 13:58:30.314985 (+ 80us) write_op.cc:276] Timestamp: P: 1777643907051487 usec, L: 0
0501 13:58:30.315438 (+ 453us) write_op.cc:270] Start()
0501 13:58:30.315514 (+ 76us) write_op.cc:276] Timestamp: P: 1777643907101096 usec, L: 0
0501 13:58:30.315982 (+ 468us) write_op.cc:270] Start()
0501 13:58:30.316064 (+ 82us) write_op.cc:276] Timestamp: P: 1777643907208385 usec, L: 0
0501 13:58:30.316514 (+ 450us) write_op.cc:270] Start()
0501 13:58:30.316591 (+ 77us) write_op.cc:276] Timestamp: P: 1777643907247218 usec, L: 0
0501 13:58:30.317041 (+ 450us) write_op.cc:270] Start()
0501 13:58:30.317116 (+ 75us) write_op.cc:276] Timestamp: P: 1777643907260336 usec, L: 0
0501 13:58:30.317852 (+ 736us) write_op.cc:270] Start()
0501 13:58:30.317930 (+ 78us) write_op.cc:276] Timestamp: P: 1777643907318736 usec, L: 0
0501 13:58:30.318912 (+ 982us) write_op.cc:270] Start()
0501 13:58:30.318998 (+ 86us) write_op.cc:276] Timestamp: P: 1777643907382414 usec, L: 0
0501 13:58:30.319467 (+ 469us) write_op.cc:270] Start()
0501 13:58:30.319558 (+ 91us) write_op.cc:276] Timestamp: P: 1777643907482603 usec, L: 0
0501 13:58:30.320037 (+ 479us) write_op.cc:270] Start()
0501 13:58:30.320116 (+ 79us) write_op.cc:276] Timestamp: P: 1777643907529722 usec, L: 0
0501 13:58:30.320578 (+ 462us) write_op.cc:270] Start()
0501 13:58:30.320655 (+ 77us) write_op.cc:276] Timestamp: P: 1777643907547030 usec, L: 0
0501 13:58:30.321093 (+ 438us) write_op.cc:270] Start()
0501 13:58:30.321168 (+ 75us) write_op.cc:276] Timestamp: P: 1777643907622841 usec, L: 0
0501 13:58:30.321605 (+ 437us) write_op.cc:270] Start()
0501 13:58:30.321681 (+ 76us) write_op.cc:276] Timestamp: P: 1777643907699613 usec, L: 0
0501 13:58:30.322137 (+ 456us) write_op.cc:270] Start()
0501 13:58:30.322264 (+ 127us) write_op.cc:276] Timestamp: P: 1777643907761244 usec, L: 0
0501 13:58:30.322796 (+ 532us) write_op.cc:270] Start()
0501 13:58:30.322886 (+ 90us) write_op.cc:276] Timestamp: P: 1777643907800954 usec, L: 0
0501 13:58:30.323339 (+ 453us) write_op.cc:270] Start()
0501 13:58:30.323421 (+ 82us) write_op.cc:276] Timestamp: P: 1777643907885357 usec, L: 0
0501 13:58:30.323894 (+ 473us) write_op.cc:270] Start()
0501 13:58:30.323971 (+ 77us) write_op.cc:276] Timestamp: P: 1777643908027647 usec, L: 0
0501 13:58:30.324432 (+ 461us) write_op.cc:270] Start()
0501 13:58:30.324509 (+ 77us) write_op.cc:276] Timestamp: P: 1777643908042917 usec, L: 0
0501 13:58:30.325063 (+ 554us) write_op.cc:270] Start()
0501 13:58:30.325143 (+ 80us) write_op.cc:276] Timestamp: P: 1777643908064104 usec, L: 0
0501 13:58:30.325625 (+ 482us) write_op.cc:270] Start()
0501 13:58:30.325706 (+ 81us) write_op.cc:276] Timestamp: P: 1777643908137238 usec, L: 0
0501 13:58:30.326177 (+ 471us) write_op.cc:270] Start()
0501 13:58:30.326266 (+ 89us) write_op.cc:276] Timestamp: P: 1777643908192652 usec, L: 0
0501 13:58:30.326753 (+ 487us) write_op.cc:270] Start()
0501 13:58:30.326837 (+ 84us) write_op.cc:276] Timestamp: P: 1777643908267270 usec, L: 0
0501 13:58:30.327298 (+ 461us) write_op.cc:270] Start()
0501 13:58:30.327373 (+ 75us) write_op.cc:276] Timestamp: P: 1777643908374930 usec, L: 0
0501 13:58:30.327828 (+ 455us) write_op.cc:270] Start()
0501 13:58:30.327916 (+ 88us) write_op.cc:276] Timestamp: P: 1777643908520620 usec, L: 0
0501 13:58:30.328364 (+ 448us) write_op.cc:270] Start()
0501 13:58:30.329376 (+ 1012us) write_op.cc:276] Timestamp: P: 1777643908623113 usec, L: 0
0501 13:58:30.329867 (+ 491us) write_op.cc:270] Start()
0501 13:58:30.330018 (+ 151us) write_op.cc:276] Timestamp: P: 1777643908701651 usec, L: 0
0501 13:58:30.330616 (+ 598us) write_op.cc:270] Start()
0501 13:58:30.330699 (+ 83us) write_op.cc:276] Timestamp: P: 1777643908758497 usec, L: 0
0501 13:58:30.331178 (+ 479us) write_op.cc:270] Start()
0501 13:58:30.335215 (+ 4037us) write_op.cc:276] Timestamp: P: 1777643908788004 usec, L: 0
0501 13:58:30.343805 (+ 8590us) log.cc:844] Serialized 33414 byte log entry
0501 13:58:30.370033 (+ 26228us) raft_consensus.cc:1645] Marking committed up to 68
0501 13:58:30.370332 (+ 299us) log.cc:844] Serialized 14 byte log entry
0501 13:58:30.411727 (+ 41395us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.411753 (+ 26us) write_op.cc:454] Released schema lock
0501 13:58:30.433396 (+ 21643us) raft_consensus.cc:1697] Filling consensus response to leader.
0501 13:58:30.438379 (+ 4983us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b5400030554. stack: 00007f4936204c83 00007f4936204b80 00007f4935d75574 000000000050e094 000000000050e029 00007f493d254470 00007f493d2f331a 00007f493f1398ac 00007f493f141ebc 00007f493f141e68 00007f493f141de8 00007f493f141da3 00007f493f14104c 00007f493d730ad6 00007f493d730a68 00007f493d30997a
0501 13:58:30.444334 (+ 5955us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0501 13:58:30.444535 (+ 201us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b5400030554. stack: 00007f4936204c83 00007f4936204b80 00007f4935d75574 000000000050e094 000000000050e029 00007f493d254470 00007f493d2f331a 00007f493f1398ac 00007f493f141ebc 00007f493f141e68 00007f493f141de8 00007f493f141da3 00007f493f14104c 00007f493d730ad6 00007f493d730a68 00007f493d30997a
0501 13:58:30.466418 (+ 21883us) raft_consensus.cc:1685] finished
0501 13:58:30.466456 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0501 13:58:30.466880 (+ 424us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:30.158443 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:30.158929 (+ 486us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:30.158958 (+ 29us) write_op.cc:435] Acquired schema lock
0501 13:58:30.158975 (+ 17us) tablet.cc:662] Decoding operations
0501 13:58:30.159267 (+ 292us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:30.184525 (+ 25258us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:30.184560 (+ 35us) tablet.cc:685] Acquiring locks for 1 operations
0501 13:58:30.184752 (+ 192us) tablet.cc:701] Row locks acquired
0501 13:58:30.184762 (+ 10us) write_op.cc:260] PREPARE: finished
0501 13:58:30.402141 (+217379us) op_driver.cc:464] REPLICATION: finished
0501 13:58:30.409680 (+ 7539us) write_op.cc:301] APPLY: starting
0501 13:58:30.409748 (+ 68us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:30.409931 (+ 183us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:30.409953 (+ 22us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:30.410219 (+ 266us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:30.410507 (+ 288us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:30.410631 (+ 124us) write_op.cc:312] APPLY: finished
0501 13:58:30.410943 (+ 312us) log.cc:844] Serialized 22 byte log entry
0501 13:58:30.411222 (+ 279us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.411287 (+ 65us) write_op.cc:454] Released schema lock
0501 13:58:30.411326 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0501 13:58:30.185157 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:30.185590 (+ 433us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:30.185614 (+ 24us) write_op.cc:435] Acquired schema lock
0501 13:58:30.185628 (+ 14us) tablet.cc:662] Decoding operations
0501 13:58:30.185951 (+ 323us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:30.186030 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:30.186058 (+ 28us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:30.186264 (+ 206us) tablet.cc:701] Row locks acquired
0501 13:58:30.186280 (+ 16us) write_op.cc:260] PREPARE: finished
0501 13:58:30.412027 (+225747us) op_driver.cc:464] REPLICATION: finished
0501 13:58:30.430697 (+ 18670us) write_op.cc:301] APPLY: starting
0501 13:58:30.430786 (+ 89us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:30.430897 (+ 111us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:30.430917 (+ 20us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:30.431275 (+ 358us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:30.431419 (+ 144us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:30.431527 (+ 108us) write_op.cc:312] APPLY: finished
0501 13:58:30.435855 (+ 4328us) log.cc:844] Serialized 28 byte log entry
0501 13:58:30.436129 (+ 274us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.442248 (+ 6119us) write_op.cc:454] Released schema lock
0501 13:58:30.442287 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:30.442546 (+ 259us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.442567 (+ 21us) write_op.cc:454] Released schema lock
Related trace 'op':
0501 13:58:30.186538 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:30.186957 (+ 419us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:30.186980 (+ 23us) write_op.cc:435] Acquired schema lock
0501 13:58:30.186991 (+ 11us) tablet.cc:662] Decoding operations
0501 13:58:30.190826 (+ 3835us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:30.190887 (+ 61us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:30.190912 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0501 13:58:30.412648 (+221736us) op_driver.cc:464] REPLICATION: finished
0501 13:58:30.436266 (+ 23618us) lock_manager.cc:598] Waited 245032us for lock on <redacted> (key hash 5858686009825915071)
0501 13:58:30.436318 (+ 52us) tablet.cc:701] Row locks acquired
0501 13:58:30.436339 (+ 21us) write_op.cc:260] PREPARE: finished
0501 13:58:30.442827 (+ 6488us) write_op.cc:301] APPLY: starting
0501 13:58:30.442884 (+ 57us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:30.442971 (+ 87us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:30.442991 (+ 20us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:30.443160 (+ 169us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:30.443366 (+ 206us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:58:30.443413 (+ 47us) write_op.cc:312] APPLY: finished
0501 13:58:30.443593 (+ 180us) log.cc:844] Serialized 22 byte log entry
0501 13:58:30.443811 (+ 218us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.443871 (+ 60us) write_op.cc:454] Released schema lock
0501 13:58:30.443900 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:30.479594 (+ 35694us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.479622 (+ 28us) write_op.cc:454] Released schema lock
Related trace 'op':
0501 13:58:30.412770 (+ 0us) op_driver.cc:464] REPLICATION: finished
0501 13:58:30.479921 (+ 67151us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:30.480344 (+ 423us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:30.480378 (+ 34us) write_op.cc:435] Acquired schema lock
0501 13:58:30.480393 (+ 15us) tablet.cc:662] Decoding operations
0501 13:58:30.480711 (+ 318us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:30.480780 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:30.480807 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:30.481015 (+ 208us) tablet.cc:701] Row locks acquired
0501 13:58:30.481032 (+ 17us) write_op.cc:260] PREPARE: finished
0501 13:58:30.482416 (+ 1384us) write_op.cc:301] APPLY: starting
0501 13:58:30.482500 (+ 84us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:30.482607 (+ 107us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:30.482624 (+ 17us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:30.482987 (+ 363us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:30.483233 (+ 246us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:58:30.483341 (+ 108us) write_op.cc:312] APPLY: finished
0501 13:58:30.483609 (+ 268us) log.cc:844] Serialized 28 byte log entry
0501 13:58:30.483845 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.483911 (+ 66us) write_op.cc:454] Released schema lock
0501 13:58:30.483945 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:30.484137 (+ 192us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.484160 (+ 23us) write_op.cc:454] Released schema lock
Related trace 'op':
0501 13:58:30.412873 (+ 0us) op_driver.cc:464] REPLICATION: finished
0501 13:58:30.481392 (+ 68519us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:30.481809 (+ 417us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:30.481833 (+ 24us) write_op.cc:435] Acquired schema lock
0501 13:58:30.481848 (+ 15us) tablet.cc:662] Decoding operations
0501 13:58:30.489005 (+ 7157us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:30.489075 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:30.489102 (+ 27us) tablet.cc:685] Acquiring locks for 1 operations
0501 13:58:30.489268 (+ 166us) tablet.cc:701] Row locks acquired
0501 13:58:30.489286 (+ 18us) write_op.cc:260] PREPARE: finished
0501 13:58:30.490274 (+ 988us) write_op.cc:301] APPLY: starting
0501 13:58:30.490333 (+ 59us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:30.490436 (+ 103us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:30.490454 (+ 18us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:30.490686 (+ 232us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:30.492955 (+ 2269us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:58:30.493116 (+ 161us) write_op.cc:312] APPLY: finished
0501 13:58:30.493320 (+ 204us) log.cc:844] Serialized 22 byte log entry
0501 13:58:30.497003 (+ 3683us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.497081 (+ 78us) write_op.cc:454] Released schema lock
0501 13:58:30.497115 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:30.497336 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.497363 (+ 27us) write_op.cc:454] Released schema lock
Related trace 'op':
0501 13:58:30.412973 (+ 0us) op_driver.cc:464] REPLICATION: finished
0501 13:58:30.489640 (+ 76667us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:30.494367 (+ 4727us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:30.494394 (+ 27us) write_op.cc:435] Acquired schema lock
0501 13:58:30.494409 (+ 15us) tablet.cc:662] Decoding operations
0501 13:58:30.494777 (+ 368us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:30.494831 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:30.494858 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:30.499068 (+ 4210us) tablet.cc:701] Row locks acquired
0501 13:58:30.499089 (+ 21us) write_op.cc:260] PREPARE: finished
0501 13:58:30.500809 (+ 1720us) write_op.cc:301] APPLY: starting
0501 13:58:30.500865 (+ 56us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:30.500956 (+ 91us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:30.500974 (+ 18us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:30.501255 (+ 281us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:30.501456 (+ 201us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:58:30.501560 (+ 104us) write_op.cc:312] APPLY: finished
0501 13:58:30.501765 (+ 205us) log.cc:844] Serialized 28 byte log entry
0501 13:58:30.507733 (+ 5968us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.507818 (+ 85us) write_op.cc:454] Released schema lock
0501 13:58:30.507860 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:30.508067 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.508091 (+ 24us) write_op.cc:454] Released schema lock
Related trace 'op':
0501 13:58:30.413067 (+ 0us) op_driver.cc:464] REPLICATION: finished
0501 13:58:30.499507 (+ 86440us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:30.499986 (+ 479us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:30.500019 (+ 33us) write_op.cc:435] Acquired schema lock
0501 13:58:30.500037 (+ 18us) tablet.cc:662] Decoding operations
0501 13:58:30.500354 (+ 317us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:30.500420 (+ 66us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:30.500444 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0501 13:58:30.508332 (+ 7888us) tablet.cc:701] Row locks acquired
0501 13:58:30.508352 (+ 20us) write_op.cc:260] PREPARE: finished
0501 13:58:30.509999 (+ 1647us) write_op.cc:301] APPLY: starting
0501 13:58:30.510057 (+ 58us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:30.510191 (+ 134us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:30.510210 (+ 19us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:30.510333 (+ 123us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:30.510491 (+ 158us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:58:30.510596 (+ 105us) write_op.cc:312] APPLY: finished
0501 13:58:30.510800 (+ 204us) log.cc:844] Serialized 22 byte log entry
0501 13:58:30.511519 (+ 719us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.511582 (+ 63us) write_op.cc:454] Released schema lock
0501 13:58:30.511611 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:30.511801 (+ 190us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.511822 (+ 21us) write_op.cc:454] Released schema lock
Related trace 'op':
0501 13:58:30.413168 (+ 0us) op_driver.cc:464] REPLICATION: finished
0501 13:58:30.508694 (+ 95526us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:30.509158 (+ 464us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:30.509186 (+ 28us) write_op.cc:435] Acquired schema lock
0501 13:58:30.509202 (+ 16us) tablet.cc:662] Decoding operations
0501 13:58:30.509531 (+ 329us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:30.509590 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:30.509614 (+ 24us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:30.512019 (+ 2405us) tablet.cc:701] Row locks acquired
0501 13:58:30.512035 (+ 16us) write_op.cc:260] PREPARE: finished
0501 13:58:30.513483 (+ 1448us) write_op.cc:301] APPLY: starting
0501 13:58:30.513532 (+ 49us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:30.513622 (+ 90us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:30.513641 (+ 19us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:30.513928 (+ 287us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:30.514131 (+ 203us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:58:30.514237 (+ 106us) write_op.cc:312] APPLY: finished
0501 13:58:30.514411 (+ 174us) log.cc:844] Serialized 28 byte log entry
0501 13:58:30.518115 (+ 3704us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.518185 (+ 70us) write_op.cc:454] Released schema lock
0501 13:58:30.518219 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:30.518407 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.518428 (+ 21us) write_op.cc:454] Released schema lock
Related trace 'op':
0501 13:58:30.413265 (+ 0us) op_driver.cc:464] REPLICATION: finished
0501 13:58:30.512360 (+ 99095us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:30.512812 (+ 452us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:30.512837 (+ 25us) write_op.cc:435] Acquired schema lock
0501 13:58:30.512854 (+ 17us) tablet.cc:662] Decoding operations
0501 13:58:30.513129 (+ 275us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:30.513181 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:30.513207 (+ 26us) tablet.cc:685] Acquiring locks for 1 operations
0501 13:58:30.518686 (+ 5479us) tablet.cc:701] Row locks acquired
0501 13:58:30.518703 (+ 17us) write_op.cc:260] PREPARE: finished
0501 13:58:30.519812 (+ 1109us) write_op.cc:301] APPLY: starting
0501 13:58:30.519872 (+ 60us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:30.519961 (+ 89us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:30.519977 (+ 16us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:30.520099 (+ 122us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:30.520233 (+ 134us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 13:58:30.520339 (+ 106us) write_op.cc:312] APPLY: finished
0501 13:58:30.520517 (+ 178us) log.cc:844] Serialized 22 byte log entry
0501 13:58:30.521930 (+ 1413us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.521986 (+ 56us) write_op.cc:454] Released schema lock
0501 13:58:30.522017 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:30.522208 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.522231 (+ 23us) write_op.cc:454] Released schema lock
Related trace 'op':
0501 13:58:30.413369 (+ 0us) op_driver.cc:464] REPLICATION: finished
0501 13:58:30.519061 (+105692us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:30.519496 (+ 435us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:30.519524 (+ 28us) write_op.cc:435] Acquired schema lock
0501 13:58:30.519540 (+ 16us) tablet.cc:662] Decoding operations
0501 13:58:30.521272 (+ 1732us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:30.521330 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:30.521362 (+ 32us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:30.521570 (+ 208us) tablet.cc:701] Row locks acquired
0501 13:58:30.521587 (+ 17us) write_op.cc:260] PREPARE: finished
0501 13:58:30.522499 (+ 912us) write_op.cc:301] APPLY: starting
0501 13:58:30.522554 (+ 55us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:30.522649 (+ 95us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:30.522669 (+ 20us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:30.523025 (+ 356us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:30.523159 (+ 134us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:30.523285 (+ 126us) write_op.cc:312] APPLY: finished
0501 13:58:30.523482 (+ 197us) log.cc:844] Serialized 28 byte log entry
0501 13:58:30.523692 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:30.523764 (+ 72us) write_op.cc:454] Released schema lock
0501 13:58:30.523796 (+ 32us) write_op.cc:341] FINISH: Updating metr
I20260501 13:58:30.698501 3163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 6 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:30.714598 3936 tablet_bootstrap.cc:492] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:30.715113 3936 ts_tablet_manager.cc:1403] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.059s user 0.010s sys 0.002s
I20260501 13:58:30.717654 3936 raft_consensus.cc:359] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.718554 3936 raft_consensus.cc:385] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:30.718899 3936 raft_consensus.cc:740] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:30.719643 3936 consensus_queue.cc:260] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.735622 3936 ts_tablet_manager.cc:1434] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260501 13:58:30.756887 4008 raft_consensus.cc:493] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:30.757354 4008 raft_consensus.cc:515] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.759310 4008 leader_election.cc:290] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:30.761106 3514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034a29eda5fe4b43ac6dcf3616299e5d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:30.761742 3514 raft_consensus.cc:2468] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:30.763095 3609 leader_election.cc:304] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:30.765950 3967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:30.767215 3967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: LEADER
I20260501 13:58:30.768115 4008 raft_consensus.cc:2804] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:30.768589 4008 raft_consensus.cc:493] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:30.769030 4008 raft_consensus.cc:3060] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:30.778080 4194 raft_consensus.cc:493] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:30.778862 4008 raft_consensus.cc:515] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.778823 4194 raft_consensus.cc:515] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.780680 4194 leader_election.cc:290] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:30.781545 3514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034a29eda5fe4b43ac6dcf3616299e5d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:30.782090 3514 raft_consensus.cc:3060] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:30.782706 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034a29eda5fe4b43ac6dcf3616299e5d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:30.783483 3673 raft_consensus.cc:2393] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c239ccbdc554619a237d78e958ffbc2 in current term 1: Already voted for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in this term.
I20260501 13:58:30.789294 3947 raft_consensus.cc:2749] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:30.790652 3514 raft_consensus.cc:2468] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:30.791805 3609 leader_election.cc:304] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:30.792843 4163 raft_consensus.cc:2804] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:30.793304 4163 raft_consensus.cc:697] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:30.794063 4163 consensus_queue.cc:237] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.824209 3494 tablet_service.cc:1511] Processing CreateTablet for tablet 7654c30b114a40aca4c79ca050e2c877 (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:30.824820 4008 leader_election.cc:290] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:30.824787 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034a29eda5fe4b43ac6dcf3616299e5d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:30.825218 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7654c30b114a40aca4c79ca050e2c877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:30.825399 3822 raft_consensus.cc:2468] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:30.825506 3237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 6 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:30.827575 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034a29eda5fe4b43ac6dcf3616299e5d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:30.828148 3821 raft_consensus.cc:3060] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:30.827957 4037 leader_election.cc:304] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b0f8911dd5d431d9d8d3124a4456262; no voters: 0a1d2cd221924677a3ef81b081e5e016, 15dd70d7c5494c3c911f0a780dc9a2af
I20260501 13:58:30.829274 4179 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260501 13:58:30.835184 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 7654c30b114a40aca4c79ca050e2c877 (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:30.835834 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 7654c30b114a40aca4c79ca050e2c877 (DEFAULT_TABLE table=default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:30.836215 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7654c30b114a40aca4c79ca050e2c877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:30.836491 3514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034a29eda5fe4b43ac6dcf3616299e5d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:30.836800 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7654c30b114a40aca4c79ca050e2c877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:30.837930 3514 raft_consensus.cc:2393] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c239ccbdc554619a237d78e958ffbc2 in current term 1: Already voted for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in this term.
I20260501 13:58:30.839152 3757 leader_election.cc:304] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:30.843067 3821 raft_consensus.cc:2468] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:30.850159 4194 raft_consensus.cc:2749] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:30.854609 3936 tablet_bootstrap.cc:492] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:30.860459 3936 tablet_bootstrap.cc:654] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:30.874740 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 02ffd79e183e46489b50e1c7aecee10f with delete_type TABLET_DATA_DELETED (replaced by tablet 77b9318c50eb4b4ba9a7d124384b6f98) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:30.875488 3941 tablet_replica.cc:333] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:30.876166 3941 raft_consensus.cc:2243] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:30.876590 3941 raft_consensus.cc:2272] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:30.877701 3227 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 034a29eda5fe4b43ac6dcf3616299e5d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:30.880890 3947 consensus_queue.cc:1048] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:30.881351 3221 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 02ffd79e183e46489b50e1c7aecee10f
I20260501 13:58:30.882319 3935 tablet_bootstrap.cc:492] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:30.882988 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet 02ffd79e183e46489b50e1c7aecee10f with delete_type TABLET_DATA_DELETED (replaced by tablet 77b9318c50eb4b4ba9a7d124384b6f98) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:30.883203 3221 catalog_manager.cc:6741] ReplaceTablet: tablet 02ffd79e183e46489b50e1c7aecee10f deleted and replaced by tablet 77b9318c50eb4b4ba9a7d124384b6f98
I20260501 13:58:30.885038 3941 ts_tablet_manager.cc:1916] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:30.886536 3221 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 034a29eda5fe4b43ac6dcf3616299e5d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:30.892012 3221 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 034a29eda5fe4b43ac6dcf3616299e5d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
W20260501 13:58:30.893982 3227 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 034a29eda5fe4b43ac6dcf3616299e5d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 034a29eda5fe4b43ac6dcf3616299e5d consensus state has no leader
I20260501 13:58:30.895056 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet be144bbf949844539848011a9963ac2d from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:30.905435 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 02ffd79e183e46489b50e1c7aecee10f with delete_type TABLET_DATA_DELETED (replaced by tablet 77b9318c50eb4b4ba9a7d124384b6f98) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:30.906248 3943 tablet_replica.cc:333] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:30.906927 3943 raft_consensus.cc:2243] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:30.907371 3943 raft_consensus.cc:2272] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:30.909029 3943 ts_tablet_manager.cc:1916] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:30.908281 3941 ts_tablet_manager.cc:1929] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:30.909657 3941 log.cc:1199] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/02ffd79e183e46489b50e1c7aecee10f
I20260501 13:58:30.913389 3935 tablet_bootstrap.cc:654] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:30.916594 4021 tablet_bootstrap.cc:492] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:30.922752 3941 ts_tablet_manager.cc:1950] T 02ffd79e183e46489b50e1c7aecee10f P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:30.925905 4021 tablet_bootstrap.cc:654] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:30.930003 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet 02ffd79e183e46489b50e1c7aecee10f with delete_type TABLET_DATA_DELETED (replaced by tablet 77b9318c50eb4b4ba9a7d124384b6f98) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:30.930790 3942 tablet_replica.cc:333] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:30.931469 3942 raft_consensus.cc:2243] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:30.932157 3942 pending_rounds.cc:70] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: Trying to abort 1 pending ops.
I20260501 13:58:30.932387 3942 pending_rounds.cc:77] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229457245286400 op_type: NO_OP noop_request { }
I20260501 13:58:30.932688 3942 raft_consensus.cc:2889] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:30.932996 3942 raft_consensus.cc:2272] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:30.934759 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 02ffd79e183e46489b50e1c7aecee10f (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
W20260501 13:58:30.935446 3214 catalog_manager.cc:4969] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 02ffd79e183e46489b50e1c7aecee10f because the tablet was not found. No further retry: Not found: Tablet not found: 02ffd79e183e46489b50e1c7aecee10f
I20260501 13:58:30.943325 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet be144bbf949844539848011a9963ac2d
I20260501 13:58:30.945770 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet be144bbf949844539848011a9963ac2d with delete_type TABLET_DATA_DELETED (replaced by tablet ba69864093504babbb00fb6170774778) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:30.946511 3941 tablet_replica.cc:333] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:30.947343 3941 raft_consensus.cc:2243] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:30.947959 3941 raft_consensus.cc:2272] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:30.948376 3935 tablet_bootstrap.cc:492] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:30.949313 3935 ts_tablet_manager.cc:1403] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.003s
I20260501 13:58:30.949409 3942 ts_tablet_manager.cc:1916] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:30.949594 3941 ts_tablet_manager.cc:1916] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:58:30.954389 3757 consensus_peers.cc:597] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 -> Peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Couldn't send request to peer 9154033cea374a10a5225c53b8bcf122. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:58:30.955994 3935 raft_consensus.cc:359] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.956823 3935 raft_consensus.cc:385] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:30.957139 3935 raft_consensus.cc:740] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:30.957921 3935 consensus_queue.cc:260] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.962293 3936 tablet_bootstrap.cc:492] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:30.963109 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 02ffd79e183e46489b50e1c7aecee10f with delete_type TABLET_DATA_DELETED (replaced by tablet 77b9318c50eb4b4ba9a7d124384b6f98) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:30.966312 3935 ts_tablet_manager.cc:1434] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260501 13:58:30.969805 4021 tablet_bootstrap.cc:492] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:30.970220 4021 ts_tablet_manager.cc:1403] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.005s
I20260501 13:58:30.972683 4021 raft_consensus.cc:359] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.973291 4021 raft_consensus.cc:385] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:30.973533 4021 raft_consensus.cc:740] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:30.975409 4021 consensus_queue.cc:260] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.977993 3943 ts_tablet_manager.cc:1929] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:30.978348 3943 log.cc:1199] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/02ffd79e183e46489b50e1c7aecee10f
I20260501 13:58:30.979095 3943 ts_tablet_manager.cc:1950] T 02ffd79e183e46489b50e1c7aecee10f P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:30.979784 3936 ts_tablet_manager.cc:1403] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.125s user 0.009s sys 0.004s
I20260501 13:58:30.985396 3936 raft_consensus.cc:359] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:30.995714 3227 catalog_manager.cc:6741] ReplaceTablet: tablet be144bbf949844539848011a9963ac2d deleted and replaced by tablet ba69864093504babbb00fb6170774778
I20260501 13:58:31.001945 4021 ts_tablet_manager.cc:1434] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.031s user 0.005s sys 0.002s
I20260501 13:58:31.008186 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet be144bbf949844539848011a9963ac2d with delete_type TABLET_DATA_DELETED (replaced by tablet ba69864093504babbb00fb6170774778) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:31.009707 3936 raft_consensus.cc:385] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:31.010038 3936 raft_consensus.cc:740] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:31.007910 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet be144bbf949844539848011a9963ac2d with delete_type TABLET_DATA_DELETED (replaced by tablet ba69864093504babbb00fb6170774778) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:31.020388 3943 tablet_replica.cc:333] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:31.021150 3943 raft_consensus.cc:2243] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:31.022112 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 02ffd79e183e46489b50e1c7aecee10f (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:31.029328 3943 pending_rounds.cc:70] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: Trying to abort 1 pending ops.
I20260501 13:58:31.029991 3943 pending_rounds.cc:77] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229455334797312 op_type: NO_OP noop_request { }
I20260501 13:58:31.031206 4206 consensus_queue.cc:1048] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:58:31.030390 3943 raft_consensus.cc:2889] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:31.033524 3943 raft_consensus.cc:2272] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:31.035410 3943 ts_tablet_manager.cc:1916] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:58:31.039265 3384 hms_client.cc:305] Time spent drop HMS table: real 3.346s user 0.000s sys 0.000s
I20260501 13:58:31.040549 3228 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]
I20260501 13:58:31.049568 3221 catalog_manager.cc:5671] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:31.051090 3221 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 034a29eda5fe4b43ac6dcf3616299e5d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:31.062897 3494 tablet_service.cc:1511] Processing CreateTablet for tablet 77b9318c50eb4b4ba9a7d124384b6f98 (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.017549 3936 consensus_queue.cc:260] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.064002 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b9318c50eb4b4ba9a7d124384b6f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.069177 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 77b9318c50eb4b4ba9a7d124384b6f98 (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.069172 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 77b9318c50eb4b4ba9a7d124384b6f98 (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.070204 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b9318c50eb4b4ba9a7d124384b6f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.070206 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b9318c50eb4b4ba9a7d124384b6f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.075481 3227 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 034a29eda5fe4b43ac6dcf3616299e5d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:31.081277 3936 ts_tablet_manager.cc:1434] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.098s user 0.004s sys 0.002s
I20260501 13:58:31.095743 3941 ts_tablet_manager.cc:1929] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:31.096132 3941 log.cc:1199] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/be144bbf949844539848011a9963ac2d
I20260501 13:58:31.096781 3941 ts_tablet_manager.cc:1950] T be144bbf949844539848011a9963ac2d P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:31.097296 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet 7654c30b114a40aca4c79ca050e2c877 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:31.108227 3935 tablet_bootstrap.cc:492] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:31.114046 3935 tablet_bootstrap.cc:654] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:31.117144 3214 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet e57b93e5dadb4e5cb5077bf79bab3758 (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e57b93e5dadb4e5cb5077bf79bab3758 consensus state has no leader
I20260501 13:58:31.132814 3935 tablet_bootstrap.cc:492] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:31.086110 3942 ts_tablet_manager.cc:1929] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:31.134770 3942 log.cc:1199] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/02ffd79e183e46489b50e1c7aecee10f
I20260501 13:58:31.135555 3942 ts_tablet_manager.cc:1950] T 02ffd79e183e46489b50e1c7aecee10f P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:31.136615 3935 ts_tablet_manager.cc:1403] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.002s
I20260501 13:58:31.067849 3943 ts_tablet_manager.cc:1929] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:31.137344 3943 log.cc:1199] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/be144bbf949844539848011a9963ac2d
I20260501 13:58:31.138701 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet be144bbf949844539848011a9963ac2d (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:31.141516 3935 raft_consensus.cc:359] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.138844 3943 ts_tablet_manager.cc:1950] T be144bbf949844539848011a9963ac2d P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:31.149209 3822 raft_consensus.cc:1275] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Refusing update from remote peer 6af0cedc7f3c47299b93e6e92d8e02d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:58:31.150003 4194 raft_consensus.cc:493] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:31.150436 4194 raft_consensus.cc:515] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.152298 4194 leader_election.cc:290] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:31.152953 4021 tablet_bootstrap.cc:492] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:31.154752 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet be144bbf949844539848011a9963ac2d (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:31.156029 3221 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 7654c30b114a40aca4c79ca050e2c877
I20260501 13:58:31.156745 4008 consensus_queue.cc:1048] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:31.157323 3221 catalog_manager.cc:6741] ReplaceTablet: tablet 7654c30b114a40aca4c79ca050e2c877 deleted and replaced by tablet a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:31.159789 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 7654c30b114a40aca4c79ca050e2c877 with delete_type TABLET_DATA_DELETED (replaced by tablet a9ecb5b51ae04f32a4f07f828d563fa4) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:31.158435 4021 tablet_bootstrap.cc:654] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:31.160794 3800 tablet_service.cc:1558] Processing DeleteTablet for tablet 7654c30b114a40aca4c79ca050e2c877 with delete_type TABLET_DATA_DELETED (replaced by tablet a9ecb5b51ae04f32a4f07f828d563fa4) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:31.168236 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7654c30b114a40aca4c79ca050e2c877" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:31.169178 3673 raft_consensus.cc:2468] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:31.171542 3756 leader_election.cc:304] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:31.172925 4194 raft_consensus.cc:2804] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:31.173328 4194 raft_consensus.cc:493] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:31.173746 4194 raft_consensus.cc:3060] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:31.169916 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet 7654c30b114a40aca4c79ca050e2c877 with delete_type TABLET_DATA_DELETED (replaced by tablet a9ecb5b51ae04f32a4f07f828d563fa4) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:31.188508 4194 raft_consensus.cc:515] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.192481 3936 tablet_bootstrap.cc:492] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:31.197651 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 02ffd79e183e46489b50e1c7aecee10f (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:31.208210 3935 raft_consensus.cc:385] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:31.208513 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7654c30b114a40aca4c79ca050e2c877" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:31.208947 3943 tablet_replica.cc:333] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:31.209122 3513 raft_consensus.cc:3060] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:31.209739 3943 raft_consensus.cc:2243] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:31.210180 3943 raft_consensus.cc:2272] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:31.154949 3514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7654c30b114a40aca4c79ca050e2c877" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:31.157626 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 7654c30b114a40aca4c79ca050e2c877 with delete_type TABLET_DATA_DELETED (replaced by tablet a9ecb5b51ae04f32a4f07f828d563fa4) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:31.211966 3941 tablet_replica.cc:333] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:31.212261 4021 tablet_bootstrap.cc:492] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:31.212649 4021 ts_tablet_manager.cc:1403] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.004s
I20260501 13:58:31.208537 3935 raft_consensus.cc:740] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:31.191668 3515 raft_consensus.cc:1275] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Refusing update from remote peer 6af0cedc7f3c47299b93e6e92d8e02d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:58:31.215263 3935 consensus_queue.cc:260] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.222187 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet 77b9318c50eb4b4ba9a7d124384b6f98 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:31.222013 4204 raft_consensus.cc:493] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:31.222621 4204 raft_consensus.cc:515] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.225299 4204 leader_election.cc:290] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:31.227571 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b9318c50eb4b4ba9a7d124384b6f98" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:31.229264 4008 consensus_queue.cc:1048] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:58:31.237957 3448 leader_election.cc:343] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:31.240166 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77b9318c50eb4b4ba9a7d124384b6f98" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:31.240489 3935 ts_tablet_manager.cc:1434] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.101s user 0.004s sys 0.001s
I20260501 13:58:31.237787 3513 raft_consensus.cc:2468] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:31.241181 3941 raft_consensus.cc:2243] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:31.243290 3941 raft_consensus.cc:2272] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:31.245225 3941 ts_tablet_manager.cc:1916] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:31.275671 4021 raft_consensus.cc:359] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.279229 3757 leader_election.cc:304] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
W20260501 13:58:31.279882 3757 raft_consensus.cc:2686] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260501 13:58:31.290477 4021 raft_consensus.cc:385] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:31.290876 4021 raft_consensus.cc:740] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:31.291671 4021 consensus_queue.cc:260] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:31.296447 3448 leader_election.cc:343] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:31.296917 3448 leader_election.cc:304] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:31.298333 4204 raft_consensus.cc:2749] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:31.308250 3220 catalog_manager.cc:2174] created HMS catalog entry for table [id=8d51c67732064782bd5d53da7178ba87]
I20260501 13:58:31.311671 4222 tablet.cc:1724] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:31.313382 3941 ts_tablet_manager.cc:1929] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:31.313735 3941 log.cc:1199] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/7654c30b114a40aca4c79ca050e2c877
I20260501 13:58:31.307017 4213 mvcc.cc:204] Tried to move back new op lower bound from 7281229459962703872 to 7281229458625548288. Current Snapshot: MvccSnapshot[applied={T|T < 7281229459962703872}]
I20260501 13:58:31.326829 4168 mvcc.cc:204] Tried to move back new op lower bound from 7281229450395664384 to 7281229430057959424. Current Snapshot: MvccSnapshot[applied={T|T < 7281229450395664384 or (T in {7281229450395664384})}]
I20260501 13:58:31.329247 4021 ts_tablet_manager.cc:1434] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.116s user 0.008s sys 0.000s
I20260501 13:58:31.314329 3941 ts_tablet_manager.cc:1950] T 7654c30b114a40aca4c79ca050e2c877 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:31.334623 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 7654c30b114a40aca4c79ca050e2c877 (table default.table_4c6e426a58b5461bbfaef6abc2ffd23f [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:31.369730 4226 tablet.cc:1724] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:31.389973 3221 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 77b9318c50eb4b4ba9a7d124384b6f98
I20260501 13:58:31.391374 3221 catalog_manager.cc:6741] ReplaceTablet: tablet 77b9318c50eb4b4ba9a7d124384b6f98 deleted and replaced by tablet 258801137bf64cdd8588cac0d48c5c7d
I20260501 13:58:31.393589 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 77b9318c50eb4b4ba9a7d124384b6f98 with delete_type TABLET_DATA_DELETED (replaced by tablet 258801137bf64cdd8588cac0d48c5c7d) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:31.394297 3941 tablet_replica.cc:333] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:31.394968 3941 raft_consensus.cc:2243] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:31.395412 3941 raft_consensus.cc:2272] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:31.401428 3941 ts_tablet_manager.cc:1916] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:31.425643 3936 tablet_bootstrap.cc:654] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:31.434672 3213 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.188:38253, user_credentials={real_user=slave}} blocked reactor thread for 66652.2us
I20260501 13:58:31.440646 3227 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 034a29eda5fe4b43ac6dcf3616299e5d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:31.444139 3221 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 034a29eda5fe4b43ac6dcf3616299e5d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:31.445878 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet af227b4c28e34aa6abac901bdfb92415 from {username='slave'} at 127.0.0.1:50472
W20260501 13:58:31.459822 3213 catalog_manager.cc:4969] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet 02ffd79e183e46489b50e1c7aecee10f because the tablet was not found. No further retry: Not found: Tablet not found: 02ffd79e183e46489b50e1c7aecee10f
I20260501 13:58:31.430346 3942 tablet_replica.cc:333] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:31.436305 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 77b9318c50eb4b4ba9a7d124384b6f98 with delete_type TABLET_DATA_DELETED (replaced by tablet 258801137bf64cdd8588cac0d48c5c7d) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:31.472676 3942 raft_consensus.cc:2243] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:31.473245 3942 raft_consensus.cc:2272] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:31.462831 4221 tablet.cc:1724] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:31.483387 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 77b9318c50eb4b4ba9a7d124384b6f98 with delete_type TABLET_DATA_DELETED (replaced by tablet 258801137bf64cdd8588cac0d48c5c7d) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:31.511443 4194 leader_election.cc:290] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:31.516489 3942 ts_tablet_manager.cc:1916] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:31.521493 3943 ts_tablet_manager.cc:1916] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:31.527709 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7654c30b114a40aca4c79ca050e2c877" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:31.528425 3673 raft_consensus.cc:3060] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:31.545729 3673 raft_consensus.cc:2468] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:31.551911 3219 catalog_manager.cc:2174] created HMS catalog entry for table [id=18ac75baaaf7485da21228e06a780ad2]
I20260501 13:58:31.571574 3220 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50472 (request call id 53) took 6282 ms. Trace:
I20260501 13:58:31.572739 3221 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet af227b4c28e34aa6abac901bdfb92415
I20260501 13:58:31.573184 3941 ts_tablet_manager.cc:1929] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:31.573511 3941 log.cc:1199] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/77b9318c50eb4b4ba9a7d124384b6f98
I20260501 13:58:31.574096 3221 catalog_manager.cc:6741] ReplaceTablet: tablet af227b4c28e34aa6abac901bdfb92415 deleted and replaced by tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:31.574141 3941 ts_tablet_manager.cc:1950] T 77b9318c50eb4b4ba9a7d124384b6f98 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:31.576162 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet af227b4c28e34aa6abac901bdfb92415 with delete_type TABLET_DATA_DELETED (replaced by tablet 19f08ae176df4a369c635a562b3dc3ba) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:31.571995 3220 rpcz_store.cc:276] 0501 13:58:25.289170 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:25.289330 (+ 160us) service_pool.cc:224] Handling call
0501 13:58:26.848059 (+1558729us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:26.849893 (+ 1834us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:31.308113 (+4458220us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:31.570647 (+262534us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:31.571101 (+ 454us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:31.571480 (+ 379us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:31.436679 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:31.437233 (+ 554us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:31.437259 (+ 26us) write_op.cc:435] Acquired schema lock
0501 13:58:31.437271 (+ 12us) tablet.cc:662] Decoding operations
0501 13:58:31.437539 (+ 268us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:31.437579 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:31.437600 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:31.463765 (+ 26165us) tablet.cc:701] Row locks acquired
0501 13:58:31.463780 (+ 15us) write_op.cc:260] PREPARE: finished
0501 13:58:31.463900 (+ 120us) write_op.cc:270] Start()
0501 13:58:31.463979 (+ 79us) write_op.cc:276] Timestamp: P: 1777643911463851 usec, L: 0
0501 13:58:31.464004 (+ 25us) op_driver.cc:348] REPLICATION: starting
0501 13:58:31.466986 (+ 2982us) log.cc:844] Serialized 432 byte log entry
0501 13:58:31.564475 (+ 97489us) op_driver.cc:464] REPLICATION: finished
0501 13:58:31.569061 (+ 4586us) write_op.cc:301] APPLY: starting
0501 13:58:31.569115 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:31.569215 (+ 100us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:31.569299 (+ 84us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:31.569722 (+ 423us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:31.569864 (+ 142us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:31.569923 (+ 59us) write_op.cc:312] APPLY: finished
0501 13:58:31.570127 (+ 204us) log.cc:844] Serialized 28 byte log entry
0501 13:58:31.570272 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:31.570339 (+ 67us) write_op.cc:454] Released schema lock
0501 13:58:31.570370 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:31.570670 (+ 300us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:31.570713 (+ 43us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":4189373,"Hive Metastore.run_cpu_time_us":682,"Hive Metastore.run_wall_time_us":267528,"child_traces":[["op",{"apply.queue_time_us":4416,"apply.run_cpu_time_us":1581,"apply.run_wall_time_us":1581,"num_ops":2,"prepare.queue_time_us":126654,"prepare.run_cpu_time_us":2196,"prepare.run_wall_time_us":55580,"raft.queue_time_us":3216,"raft.run_cpu_time_us":733,"raft.run_wall_time_us":732,"replication_time_us":100367}]]}
I20260501 13:58:31.578442 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet af227b4c28e34aa6abac901bdfb92415 with delete_type TABLET_DATA_DELETED (replaced by tablet 19f08ae176df4a369c635a562b3dc3ba) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:31.581015 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet af227b4c28e34aa6abac901bdfb92415 with delete_type TABLET_DATA_DELETED (replaced by tablet 19f08ae176df4a369c635a562b3dc3ba) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:31.594312 3227 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee] alter complete (version 1)
I20260501 13:58:31.601161 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 77b9318c50eb4b4ba9a7d124384b6f98 (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:31.601532 3941 tablet_replica.cc:333] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:31.602157 3941 raft_consensus.cc:2243] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:31.603226 3941 raft_consensus.cc:2272] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:31.607851 3220 master_service.cc:954] ReplaceTablet: received request to replace tablet 034a29eda5fe4b43ac6dcf3616299e5d from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:31.610999 3221 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 034a29eda5fe4b43ac6dcf3616299e5d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]). Expected version 1 got 0
I20260501 13:58:31.630854 3219 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50472 (request call id 52) took 6463 ms. Trace:
I20260501 13:58:31.631230 3219 rpcz_store.cc:276] 0501 13:58:25.167222 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:25.167476 (+ 254us) service_pool.cc:224] Handling call
0501 13:58:26.844333 (+1676857us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:26.846132 (+ 1799us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:31.551751 (+4705619us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:31.630053 (+ 78302us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:31.630393 (+ 340us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:31.630785 (+ 392us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:31.594659 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:31.595113 (+ 454us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:31.595143 (+ 30us) write_op.cc:435] Acquired schema lock
0501 13:58:31.595155 (+ 12us) tablet.cc:662] Decoding operations
0501 13:58:31.595419 (+ 264us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:31.595456 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:31.595478 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:31.595661 (+ 183us) tablet.cc:701] Row locks acquired
0501 13:58:31.595673 (+ 12us) write_op.cc:260] PREPARE: finished
0501 13:58:31.595793 (+ 120us) write_op.cc:270] Start()
0501 13:58:31.595872 (+ 79us) write_op.cc:276] Timestamp: P: 1777643911595742 usec, L: 0
0501 13:58:31.595886 (+ 14us) op_driver.cc:348] REPLICATION: starting
0501 13:58:31.598975 (+ 3089us) log.cc:844] Serialized 432 byte log entry
0501 13:58:31.604365 (+ 5390us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b5400030aa0. stack: 00007fc5c69e2c83 00007fc5c69e2b80 00007fc5c6553574 000000000050e094 000000000050e029 00007fc5cda32470 00007fc5cda3fa98 00007fc5cda3f6da 00007fc5cdac3d45 00007fc5cdac41e9 00007fc5cdf369f1 00007fc5cdf361c8 00007fc5cdf3ae61 00007fc5cdf3ae19 00007fc5cdf3ada9 00007fc5cdf3ad71
0501 13:58:31.614378 (+ 10013us) op_driver.cc:464] REPLICATION: finished
0501 13:58:31.615341 (+ 963us) write_op.cc:301] APPLY: starting
0501 13:58:31.615396 (+ 55us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:31.615498 (+ 102us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:31.615515 (+ 17us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:31.623088 (+ 7573us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b5400030554. stack: 00007fc5c69e2c83 00007fc5c69e2b80 00007fc5c6553574 000000000050e094 000000000050e029 00007fc5cda31951 00007fc5cdac4208 00007fc5cdf369f1 00007fc5cdf361c8 00007fc5cdf3ae61 00007fc5cdf3ae19 00007fc5cdf3ada9 00007fc5cdf3ad71 00007fc5cdf39f9d 00000000005b3bc4 00000000005b39f9
0501 13:58:31.628719 (+ 5631us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:31.628862 (+ 143us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:31.628905 (+ 43us) write_op.cc:312] APPLY: finished
0501 13:58:31.629098 (+ 193us) log.cc:844] Serialized 28 byte log entry
0501 13:58:31.629728 (+ 630us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:31.629799 (+ 71us) write_op.cc:454] Released schema lock
0501 13:58:31.629833 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:31.630052 (+ 219us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:31.630074 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":4449556,"Hive Metastore.run_cpu_time_us":637,"Hive Metastore.run_wall_time_us":243837,"child_traces":[["op",{"apply.queue_time_us":807,"apply.run_cpu_time_us":1582,"apply.run_wall_time_us":14685,"num_ops":2,"prepare.queue_time_us":40805,"prepare.run_cpu_time_us":3449,"prepare.run_wall_time_us":28558,"replication_time_us":18379,"spinlock_wait_cycles":13949184}]]}
I20260501 13:58:31.645335 3943 ts_tablet_manager.cc:1929] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:31.645864 3943 log.cc:1199] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/7654c30b114a40aca4c79ca050e2c877
I20260501 13:58:31.647821 3941 ts_tablet_manager.cc:1916] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:58:31.655119 3387 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9ecb5b51ae04f32a4f07f828d563fa4 has no consensus state
I20260501 13:58:31.660290 3387 catalog_manager.cc:2772] Deleting table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3] in response to Hive Metastore notification log event 9
I20260501 13:58:31.662837 3943 ts_tablet_manager.cc:1950] T 7654c30b114a40aca4c79ca050e2c877 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:31.675318 3942 ts_tablet_manager.cc:1929] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:31.677729 3942 log.cc:1199] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/be144bbf949844539848011a9963ac2d
I20260501 13:58:31.682205 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 7654c30b114a40aca4c79ca050e2c877 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:31.683646 3943 tablet_replica.cc:333] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:31.684340 3943 raft_consensus.cc:2243] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:31.685282 3942 ts_tablet_manager.cc:1950] T be144bbf949844539848011a9963ac2d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
W20260501 13:58:31.683211 3213 catalog_manager.cc:4969] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet 7654c30b114a40aca4c79ca050e2c877 because the tablet was not found. No further retry: Not found: Tablet not found: 7654c30b114a40aca4c79ca050e2c877
I20260501 13:58:31.690800 3943 raft_consensus.cc:2272] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:31.687749 3220 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 034a29eda5fe4b43ac6dcf3616299e5d
I20260501 13:58:31.692247 3220 catalog_manager.cc:6741] ReplaceTablet: tablet 034a29eda5fe4b43ac6dcf3616299e5d deleted and replaced by tablet 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:58:31.693704 3494 tablet_service.cc:1558] Processing DeleteTablet for tablet 034a29eda5fe4b43ac6dcf3616299e5d with delete_type TABLET_DATA_DELETED (replaced by tablet 6309ec735c424460ad5a74c6b7ec0d8a) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:31.696044 3943 ts_tablet_manager.cc:1916] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:31.697327 3942 tablet_replica.cc:333] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:31.698603 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 034a29eda5fe4b43ac6dcf3616299e5d with delete_type TABLET_DATA_DELETED (replaced by tablet 6309ec735c424460ad5a74c6b7ec0d8a) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:31.700481 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet 034a29eda5fe4b43ac6dcf3616299e5d with delete_type TABLET_DATA_DELETED (replaced by tablet 6309ec735c424460ad5a74c6b7ec0d8a) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:31.703357 3942 raft_consensus.cc:2243] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:31.703135 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 034a29eda5fe4b43ac6dcf3616299e5d with delete_type TABLET_DATA_DELETED (replaced by tablet 6309ec735c424460ad5a74c6b7ec0d8a) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:31.710790 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet be144bbf949844539848011a9963ac2d (table default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]) successfully deleted
I20260501 13:58:31.717615 3942 raft_consensus.cc:2272] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:31.723161 3942 ts_tablet_manager.cc:1916] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:31.726184 3936 tablet_bootstrap.cc:492] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:31.726652 3936 ts_tablet_manager.cc:1403] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.534s user 0.010s sys 0.006s
I20260501 13:58:31.727118 3652 tablet_service.cc:1511] Processing CreateTablet for tablet ba69864093504babbb00fb6170774778 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.728101 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba69864093504babbb00fb6170774778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.729038 3936 raft_consensus.cc:359] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.729645 3936 raft_consensus.cc:385] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:31.729898 3936 raft_consensus.cc:740] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:31.732812 3802 tablet_service.cc:1511] Processing CreateTablet for tablet ba69864093504babbb00fb6170774778 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.733954 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba69864093504babbb00fb6170774778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.735200 3494 tablet_service.cc:1511] Processing CreateTablet for tablet ba69864093504babbb00fb6170774778 (DEFAULT_TABLE table=default.table_5c8ebfa533ee4e9a94d8200053bfe4c0 [id=096f180e34c94777bb6cf87fded9c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.736155 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba69864093504babbb00fb6170774778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.738914 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 258801137bf64cdd8588cac0d48c5c7d (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.739100 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 258801137bf64cdd8588cac0d48c5c7d (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.739907 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 258801137bf64cdd8588cac0d48c5c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.739961 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 258801137bf64cdd8588cac0d48c5c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.744773 3493 tablet_service.cc:1511] Processing CreateTablet for tablet a9ecb5b51ae04f32a4f07f828d563fa4 (DEFAULT_TABLE table=default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.745898 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9ecb5b51ae04f32a4f07f828d563fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.751549 3800 tablet_service.cc:1511] Processing CreateTablet for tablet 258801137bf64cdd8588cac0d48c5c7d (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.752851 3800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 258801137bf64cdd8588cac0d48c5c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.752719 3653 tablet_service.cc:1511] Processing CreateTablet for tablet a9ecb5b51ae04f32a4f07f828d563fa4 (DEFAULT_TABLE table=default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.753881 3653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9ecb5b51ae04f32a4f07f828d563fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.772329 3801 tablet_service.cc:1511] Processing CreateTablet for tablet a9ecb5b51ae04f32a4f07f828d563fa4 (DEFAULT_TABLE table=default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.773429 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9ecb5b51ae04f32a4f07f828d563fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.780757 4021 tablet_bootstrap.cc:492] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:31.781584 3220 master_service.cc:954] ReplaceTablet: received request to replace tablet 034a29eda5fe4b43ac6dcf3616299e5d from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:31.786347 3220 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_f52f85ad11fb4023adead2ab9c5eb694"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:31.730489 3936 consensus_queue.cc:260] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.791337 3936 ts_tablet_manager.cc:1434] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.064s user 0.013s sys 0.004s
I20260501 13:58:31.795262 3936 tablet_bootstrap.cc:492] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:31.803123 3936 tablet_bootstrap.cc:654] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:31.809016 3941 ts_tablet_manager.cc:1929] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:31.809633 3941 log.cc:1199] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/af227b4c28e34aa6abac901bdfb92415
I20260501 13:58:31.810562 3387 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet ba69864093504babbb00fb6170774778
I20260501 13:58:31.810690 3936 tablet_bootstrap.cc:492] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:31.811103 3936 ts_tablet_manager.cc:1403] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260501 13:58:31.813573 3936 raft_consensus.cc:359] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.814652 3936 raft_consensus.cc:385] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:31.814620 3492 tablet_service.cc:1558] Processing DeleteTablet for tablet ba69864093504babbb00fb6170774778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:31 UTC) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:31.814903 3936 raft_consensus.cc:740] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:31.815461 3936 consensus_queue.cc:260] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.816314 3653 tablet_service.cc:1558] Processing DeleteTablet for tablet ba69864093504babbb00fb6170774778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:31 UTC) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:31.818725 4021 tablet_bootstrap.cc:654] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:31.819181 3800 tablet_service.cc:1558] Processing DeleteTablet for tablet ba69864093504babbb00fb6170774778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:31 UTC) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:31.819998 3936 ts_tablet_manager.cc:1434] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260501 13:58:31.839844 3219 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50472:
name: "default.Table_f701ba5ffc3845bf8cb5e55102e60688"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:31.854559 3941 ts_tablet_manager.cc:1950] T af227b4c28e34aa6abac901bdfb92415 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:31.857278 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet af227b4c28e34aa6abac901bdfb92415 (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:31.860625 3936 tablet_bootstrap.cc:492] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:31.866261 3936 tablet_bootstrap.cc:654] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:31.875494 4231 tablet_bootstrap.cc:492] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:31.880028 3941 tablet_replica.cc:333] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:31.880672 4021 tablet_bootstrap.cc:492] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:31.880654 3941 raft_consensus.cc:2243] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:31.881080 4021 ts_tablet_manager.cc:1403] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.101s user 0.008s sys 0.002s
I20260501 13:58:31.881238 3941 raft_consensus.cc:2272] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:31.883944 4021 raft_consensus.cc:359] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.884652 4021 raft_consensus.cc:385] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:31.884922 4021 raft_consensus.cc:740] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:31.885510 4021 consensus_queue.cc:260] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.899775 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet a9ecb5b51ae04f32a4f07f828d563fa4 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:31.909292 3943 ts_tablet_manager.cc:1929] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:31.909677 3943 log.cc:1199] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/77b9318c50eb4b4ba9a7d124384b6f98
I20260501 13:58:31.911419 3493 tablet_service.cc:1511] Processing CreateTablet for tablet 545ced72d1cd47faa827d7b00df372b7 (DEFAULT_TABLE table=default.table_1f445634a0fc4676a2e7bf208b0ea1de [id=18ac75baaaf7485da21228e06a780ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.912583 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 545ced72d1cd47faa827d7b00df372b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.883641 3941 ts_tablet_manager.cc:1916] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:31.918277 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 545ced72d1cd47faa827d7b00df372b7 (DEFAULT_TABLE table=default.table_1f445634a0fc4676a2e7bf208b0ea1de [id=18ac75baaaf7485da21228e06a780ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.919342 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 545ced72d1cd47faa827d7b00df372b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.919512 4021 ts_tablet_manager.cc:1434] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20260501 13:58:31.923156 3494 tablet_service.cc:1511] Processing CreateTablet for tablet 7a626cb9dab0463eb9b23098a8adfd7d (DEFAULT_TABLE table=default.table_ed51787806aa40069d50f787cc85ae8f [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.923322 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 545ced72d1cd47faa827d7b00df372b7 (DEFAULT_TABLE table=default.table_1f445634a0fc4676a2e7bf208b0ea1de [id=18ac75baaaf7485da21228e06a780ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.924265 3494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a626cb9dab0463eb9b23098a8adfd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.924265 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 545ced72d1cd47faa827d7b00df372b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.929462 3495 tablet_service.cc:1511] Processing CreateTablet for tablet 19f08ae176df4a369c635a562b3dc3ba (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.930639 3495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f08ae176df4a369c635a562b3dc3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.935245 3801 tablet_service.cc:1511] Processing CreateTablet for tablet 7a626cb9dab0463eb9b23098a8adfd7d (DEFAULT_TABLE table=default.table_ed51787806aa40069d50f787cc85ae8f [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.936409 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a626cb9dab0463eb9b23098a8adfd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.941776 3653 tablet_service.cc:1511] Processing CreateTablet for tablet 7a626cb9dab0463eb9b23098a8adfd7d (DEFAULT_TABLE table=default.table_ed51787806aa40069d50f787cc85ae8f [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.943023 3653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a626cb9dab0463eb9b23098a8adfd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.945292 3651 tablet_service.cc:1511] Processing CreateTablet for tablet 19f08ae176df4a369c635a562b3dc3ba (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.946350 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f08ae176df4a369c635a562b3dc3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.949908 3942 ts_tablet_manager.cc:1929] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:31.950258 3942 log.cc:1199] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/7654c30b114a40aca4c79ca050e2c877
I20260501 13:58:31.957610 3936 tablet_bootstrap.cc:492] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:31.958061 3936 ts_tablet_manager.cc:1403] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.098s user 0.011s sys 0.012s
I20260501 13:58:31.960464 3936 raft_consensus.cc:359] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.961098 3936 raft_consensus.cc:385] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:31.961369 3936 raft_consensus.cc:740] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:31.962111 3936 consensus_queue.cc:260] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:31.910343 3943 ts_tablet_manager.cc:1950] T 77b9318c50eb4b4ba9a7d124384b6f98 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:31.969638 3936 ts_tablet_manager.cc:1434] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260501 13:58:31.972561 3943 tablet_replica.cc:333] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:31.973230 3943 raft_consensus.cc:2243] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:31.973811 4021 tablet_bootstrap.cc:492] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:31.977065 3492 tablet_service.cc:1511] Processing CreateTablet for tablet 6309ec735c424460ad5a74c6b7ec0d8a (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.978994 3800 tablet_service.cc:1511] Processing CreateTablet for tablet 19f08ae176df4a369c635a562b3dc3ba (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:31.980062 3800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f08ae176df4a369c635a562b3dc3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:31.983498 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:31.984289 3936 tablet_bootstrap.cc:492] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:31.984819 3227 catalog_manager.cc:6741] ReplaceTablet: tablet a9ecb5b51ae04f32a4f07f828d563fa4 deleted and replaced by tablet d129cc8c1fc840cd9e00d4ad22ff236c
I20260501 13:58:31.997857 3936 tablet_bootstrap.cc:654] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:31.946295 4231 tablet_bootstrap.cc:654] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.007144 3942 ts_tablet_manager.cc:1950] T 7654c30b114a40aca4c79ca050e2c877 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:32.008541 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 6309ec735c424460ad5a74c6b7ec0d8a (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:32.011801 3799 tablet_service.cc:1511] Processing CreateTablet for tablet 6309ec735c424460ad5a74c6b7ec0d8a (DEFAULT_TABLE table=default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:32.012993 3799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6309ec735c424460ad5a74c6b7ec0d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:32.013866 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6309ec735c424460ad5a74c6b7ec0d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:32.016072 3798 tablet_service.cc:1558] Processing DeleteTablet for tablet a9ecb5b51ae04f32a4f07f828d563fa4 with delete_type TABLET_DATA_DELETED (replaced by tablet d129cc8c1fc840cd9e00d4ad22ff236c) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:32.020960 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 77b9318c50eb4b4ba9a7d124384b6f98 (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:32.021734 3936 tablet_bootstrap.cc:492] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:32.022153 3936 ts_tablet_manager.cc:1403] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.000s
I20260501 13:58:32.024403 3491 tablet_service.cc:1558] Processing DeleteTablet for tablet a9ecb5b51ae04f32a4f07f828d563fa4 with delete_type TABLET_DATA_DELETED (replaced by tablet d129cc8c1fc840cd9e00d4ad22ff236c) from {username='slave'} at 127.0.0.1:58038
I20260501 13:58:32.024888 3936 raft_consensus.cc:359] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.025498 3936 raft_consensus.cc:385] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.009524 3942 tablet_replica.cc:333] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:32.011396 3650 tablet_service.cc:1558] Processing DeleteTablet for tablet a9ecb5b51ae04f32a4f07f828d563fa4 with delete_type TABLET_DATA_DELETED (replaced by tablet d129cc8c1fc840cd9e00d4ad22ff236c) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:32.025736 3936 raft_consensus.cc:740] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.024894 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 7654c30b114a40aca4c79ca050e2c877 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:32.026513 3936 consensus_queue.cc:260] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.029440 3492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6309ec735c424460ad5a74c6b7ec0d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:32.037878 4021 tablet_bootstrap.cc:654] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.038795 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet ba69864093504babbb00fb6170774778 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:31.973755 3943 raft_consensus.cc:2272] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:32.061964 3943 ts_tablet_manager.cc:1916] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:58:32.065363 3609 consensus_peers.cc:597] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 -> Peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Couldn't send request to peer 9154033cea374a10a5225c53b8bcf122. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:58:32.071502 3936 ts_tablet_manager.cc:1434] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20260501 13:58:32.073539 3936 tablet_bootstrap.cc:492] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:32.085042 3936 tablet_bootstrap.cc:654] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.086941 4231 tablet_bootstrap.cc:492] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:32.087481 4231 ts_tablet_manager.cc:1403] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.212s user 0.005s sys 0.007s
I20260501 13:58:32.090142 4231 raft_consensus.cc:359] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.090898 4231 raft_consensus.cc:385] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.091179 4231 raft_consensus.cc:740] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.091959 3942 raft_consensus.cc:2243] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:32.091867 4231 consensus_queue.cc:260] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.092556 3942 raft_consensus.cc:2272] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:32.095312 3942 ts_tablet_manager.cc:1916] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:32.103998 3491 tablet_service.cc:1511] Processing CreateTablet for tablet d129cc8c1fc840cd9e00d4ad22ff236c (DEFAULT_TABLE table=default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:32.108836 4021 tablet_bootstrap.cc:492] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:32.109266 4021 ts_tablet_manager.cc:1403] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.168s user 0.006s sys 0.003s
I20260501 13:58:32.112401 3941 ts_tablet_manager.cc:1929] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:32.112777 3941 log.cc:1199] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/034a29eda5fe4b43ac6dcf3616299e5d
I20260501 13:58:32.113441 3941 ts_tablet_manager.cc:1950] T 034a29eda5fe4b43ac6dcf3616299e5d P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:32.116017 3214 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 034a29eda5fe4b43ac6dcf3616299e5d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:32.131882 4231 ts_tablet_manager.cc:1434] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20260501 13:58:32.135046 3936 tablet_bootstrap.cc:492] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:32.135481 3936 ts_tablet_manager.cc:1403] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.002s
I20260501 13:58:32.150607 4231 tablet_bootstrap.cc:492] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:32.153617 3651 tablet_service.cc:1511] Processing CreateTablet for tablet d129cc8c1fc840cd9e00d4ad22ff236c (DEFAULT_TABLE table=default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:32.143395 4021 raft_consensus.cc:359] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.154696 4021 raft_consensus.cc:385] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.154987 4021 raft_consensus.cc:740] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.155099 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d129cc8c1fc840cd9e00d4ad22ff236c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:32.155265 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d129cc8c1fc840cd9e00d4ad22ff236c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:58:32.156665 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet ba69864093504babbb00fb6170774778 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba69864093504babbb00fb6170774778 already in progress: creating tablet
I20260501 13:58:32.162779 4021 consensus_queue.cc:260] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:32.164745 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet a9ecb5b51ae04f32a4f07f828d563fa4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9ecb5b51ae04f32a4f07f828d563fa4 already in progress: creating tablet
I20260501 13:58:32.166357 3802 tablet_service.cc:1511] Processing CreateTablet for tablet d129cc8c1fc840cd9e00d4ad22ff236c (DEFAULT_TABLE table=default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:32.167466 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d129cc8c1fc840cd9e00d4ad22ff236c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:32.167443 3936 raft_consensus.cc:359] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.168110 3936 raft_consensus.cc:385] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.168368 3936 raft_consensus.cc:740] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.168955 3936 consensus_queue.cc:260] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.184211 4021 ts_tablet_manager.cc:1434] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.075s user 0.007s sys 0.000s
I20260501 13:58:32.186175 3936 ts_tablet_manager.cc:1434] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.050s user 0.005s sys 0.001s
I20260501 13:58:32.194774 4231 tablet_bootstrap.cc:654] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.195894 3943 ts_tablet_manager.cc:1929] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:32.202666 3221 master_service.cc:954] ReplaceTablet: received request to replace tablet 6309ec735c424460ad5a74c6b7ec0d8a from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:32.203153 3943 log.cc:1199] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/af227b4c28e34aa6abac901bdfb92415
I20260501 13:58:32.205353 3943 ts_tablet_manager.cc:1950] T af227b4c28e34aa6abac901bdfb92415 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:32.202296 4021 tablet_bootstrap.cc:492] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:32.206756 3936 tablet_bootstrap.cc:492] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:32.223644 4021 tablet_bootstrap.cc:654] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.231817 4194 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:32.232385 4194 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:32.233875 3227 catalog_manager.cc:5504] Ignoring report from unknown tablet a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:32.228288 3942 ts_tablet_manager.cc:1929] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:32.234640 3942 log.cc:1199] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/77b9318c50eb4b4ba9a7d124384b6f98
I20260501 13:58:32.235283 3942 ts_tablet_manager.cc:1950] T 77b9318c50eb4b4ba9a7d124384b6f98 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:32.235796 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9ecb5b51ae04f32a4f07f828d563fa4" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:32.236387 3673 raft_consensus.cc:2468] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:32.238487 3756 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:32.238871 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9ecb5b51ae04f32a4f07f828d563fa4" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:32.234316 4194 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:32.239882 4194 raft_consensus.cc:2804] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:32.240199 4194 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260501 13:58:32.240375 3757 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:32.240511 4194 raft_consensus.cc:3060] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.234411 3936 tablet_bootstrap.cc:654] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.247406 3942 tablet_replica.cc:333] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:32.248112 3942 raft_consensus.cc:2243] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:32.249300 3942 raft_consensus.cc:2272] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:32.252193 4194 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.256263 4163 raft_consensus.cc:493] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:32.256757 4163 raft_consensus.cc:515] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.259155 4163 leader_election.cc:290] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:32.262701 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "258801137bf64cdd8588cac0d48c5c7d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:32.263012 4194 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:32.255249 3943 tablet_replica.cc:333] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:32.263284 3942 ts_tablet_manager.cc:1916] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:32.263341 3515 raft_consensus.cc:2468] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:32.263880 3943 raft_consensus.cc:2243] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:32.264477 3943 raft_consensus.cc:2272] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:32.264742 3609 leader_election.cc:304] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:32.268044 3943 ts_tablet_manager.cc:1916] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:32.268452 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "258801137bf64cdd8588cac0d48c5c7d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:32.269114 3821 raft_consensus.cc:2468] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:32.270860 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9ecb5b51ae04f32a4f07f828d563fa4" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
W20260501 13:58:32.272418 3757 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:32.272778 4231 tablet_bootstrap.cc:492] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:32.273202 4231 ts_tablet_manager.cc:1403] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.124s user 0.009s sys 0.003s
I20260501 13:58:32.273653 4163 raft_consensus.cc:2804] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:32.274015 4163 raft_consensus.cc:493] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:32.274312 4163 raft_consensus.cc:3060] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.281919 3936 tablet_bootstrap.cc:492] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:32.282371 3936 ts_tablet_manager.cc:1403] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.076s user 0.009s sys 0.000s
I20260501 13:58:32.284310 4163 raft_consensus.cc:515] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.285014 3936 raft_consensus.cc:359] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.285601 3936 raft_consensus.cc:385] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.285861 3936 raft_consensus.cc:740] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.286952 3936 consensus_queue.cc:260] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.290251 3936 ts_tablet_manager.cc:1434] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
W20260501 13:58:32.294679 3212 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet a9ecb5b51ae04f32a4f07f828d563fa4 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a9ecb5b51ae04f32a4f07f828d563fa4 consensus state has no leader
I20260501 13:58:32.298868 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 77b9318c50eb4b4ba9a7d124384b6f98 (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:32.299288 3936 tablet_bootstrap.cc:492] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:32.299810 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet af227b4c28e34aa6abac901bdfb92415 (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:32.306327 4231 raft_consensus.cc:359] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.307045 4231 raft_consensus.cc:385] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.307360 4231 raft_consensus.cc:740] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.308053 4231 consensus_queue.cc:260] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.309481 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet 19f08ae176df4a369c635a562b3dc3ba from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:32.318284 3221 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:58:32.319325 4021 tablet_bootstrap.cc:492] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:32.319591 3221 catalog_manager.cc:6741] ReplaceTablet: tablet 6309ec735c424460ad5a74c6b7ec0d8a deleted and replaced by tablet c8ed945f80854c06ae817814f5070b13
I20260501 13:58:32.319729 4021 ts_tablet_manager.cc:1403] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.118s user 0.005s sys 0.007s
I20260501 13:58:32.319895 3942 ts_tablet_manager.cc:1929] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:32.329309 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet 6309ec735c424460ad5a74c6b7ec0d8a with delete_type TABLET_DATA_DELETED (replaced by tablet c8ed945f80854c06ae817814f5070b13) from {username='slave'} at 127.0.0.1:58038
W20260501 13:58:32.331202 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 6309ec735c424460ad5a74c6b7ec0d8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6309ec735c424460ad5a74c6b7ec0d8a already in progress: creating tablet
I20260501 13:58:32.320320 3942 log.cc:1199] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/af227b4c28e34aa6abac901bdfb92415
I20260501 13:58:32.334242 3936 tablet_bootstrap.cc:654] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.334925 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "258801137bf64cdd8588cac0d48c5c7d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:32.335494 3515 raft_consensus.cc:3060] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.336508 3942 ts_tablet_manager.cc:1950] T af227b4c28e34aa6abac901bdfb92415 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
W20260501 13:58:32.340969 3221 catalog_manager.cc:5504] Ignoring report from unknown tablet ba69864093504babbb00fb6170774778
I20260501 13:58:32.345479 4231 ts_tablet_manager.cc:1434] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.071s user 0.007s sys 0.000s
I20260501 13:58:32.322055 4021 raft_consensus.cc:359] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.346292 4021 raft_consensus.cc:385] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.346619 4021 raft_consensus.cc:740] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.355301 3942 tablet_replica.cc:333] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:32.356004 3942 raft_consensus.cc:2243] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:32.356863 3942 raft_consensus.cc:2272] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:58:32.360298 3221 catalog_manager.cc:5504] Ignoring report from unknown tablet ba69864093504babbb00fb6170774778
I20260501 13:58:32.369263 3515 raft_consensus.cc:2468] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:32.370774 3609 leader_election.cc:304] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:32.370975 4021 consensus_queue.cc:260] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.371613 4245 raft_consensus.cc:2804] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:32.364302 4231 tablet_bootstrap.cc:492] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:32.373843 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "258801137bf64cdd8588cac0d48c5c7d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:32.374378 3821 raft_consensus.cc:3060] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.381493 3821 raft_consensus.cc:2468] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
W20260501 13:58:32.385452 3221 catalog_manager.cc:5504] Ignoring report from unknown tablet ba69864093504babbb00fb6170774778
I20260501 13:58:32.387902 4245 raft_consensus.cc:697] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:32.388746 4245 consensus_queue.cc:237] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.382360 4021 ts_tablet_manager.cc:1434] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
W20260501 13:58:32.391258 3213 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.188:38253, user_credentials={real_user=slave}} blocked reactor thread for 57937.5us
I20260501 13:58:32.391497 4021 tablet_bootstrap.cc:492] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:32.391812 4163 leader_election.cc:290] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:32.395279 3221 catalog_manager.cc:5504] Ignoring report from unknown tablet ba69864093504babbb00fb6170774778
I20260501 13:58:32.400017 3213 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet af227b4c28e34aa6abac901bdfb92415 (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:32.402427 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 6309ec735c424460ad5a74c6b7ec0d8a with delete_type TABLET_DATA_DELETED (replaced by tablet c8ed945f80854c06ae817814f5070b13) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:32.403362 3942 ts_tablet_manager.cc:1916] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:32.404599 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 6309ec735c424460ad5a74c6b7ec0d8a with delete_type TABLET_DATA_DELETED (replaced by tablet c8ed945f80854c06ae817814f5070b13) from {username='slave'} at 127.0.0.1:46676
W20260501 13:58:32.405427 3756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:45857, user_credentials={real_user=slave}} blocked reactor thread for 167140us
I20260501 13:58:32.421435 4231 tablet_bootstrap.cc:654] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.407099 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9ecb5b51ae04f32a4f07f828d563fa4" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:32.424028 3673 raft_consensus.cc:3060] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.434088 3673 raft_consensus.cc:2468] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:32.435428 3756 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:32.435592 3226 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]
I20260501 13:58:32.436465 4243 raft_consensus.cc:2804] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:32.437311 4243 raft_consensus.cc:697] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:32.438194 4243 consensus_queue.cc:237] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.400245 3936 tablet_bootstrap.cc:492] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:32.441574 3936 ts_tablet_manager.cc:1403] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.142s user 0.011s sys 0.000s
I20260501 13:58:32.444022 3936 raft_consensus.cc:359] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:32.444666 3224 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aafd3ab3f9f44a74af41aae60cfe40fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:32.444839 3936 raft_consensus.cc:385] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.445096 3936 raft_consensus.cc:740] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.445652 3936 consensus_queue.cc:260] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:32.447711 3225 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_06e2e16b792549e68d2e01ce3e3ce8b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:32.449195 3220 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f52f85ad11fb4023adead2ab9c5eb694 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:32.451239 4021 tablet_bootstrap.cc:654] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:32.451649 3219 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f701ba5ffc3845bf8cb5e55102e60688 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:32.456369 3943 ts_tablet_manager.cc:1929] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:32.456825 3943 log.cc:1199] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/034a29eda5fe4b43ac6dcf3616299e5d
I20260501 13:58:32.457566 3943 ts_tablet_manager.cc:1950] T 034a29eda5fe4b43ac6dcf3616299e5d P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:32.465456 3223 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50472 (request call id 48) took 7500 ms. Trace:
I20260501 13:58:32.477027 3223 rpcz_store.cc:276] 0501 13:58:24.965140 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:24.965290 (+ 150us) service_pool.cc:224] Handling call
0501 13:58:24.984710 (+ 19420us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b5400030554. stack: 00007fc5c69e2c83 00007fc5c69e2b80 00007fc5c6553574 000000000050e094 000000000050e029 00007fc5cda32470 00007fc5cdad131a 00007fc5cf7e546c 00007fc5cf8e0fb6 00007fc5cd4f6b24 00007fc5cd4f6ab2 00007fc5cd4f69e1 00007fc5cd4f695c 00007fc5cd4f5c12 00007fc5cb7bbbcc 00007fc5cb7bb006
0501 13:58:26.761029 (+1776319us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:32.450987 (+5689958us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:32.464320 (+ 13333us) catalog_manager.cc:3725] Apply alter schema
0501 13:58:32.465272 (+ 952us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":15610,"Hive Metastore.run_cpu_time_us":1802,"Hive Metastore.run_wall_time_us":477740,"spinlock_wait_cycles":41467648}
I20260501 13:58:32.477916 3936 ts_tablet_manager.cc:1434] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20260501 13:58:32.478902 3936 tablet_bootstrap.cc:492] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
W20260501 13:58:32.481839 3223 catalog_manager.cc:5504] Ignoring report from unknown tablet a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:32.480046 3387 catalog_manager.cc:2772] Deleting table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee] in response to Hive Metastore notification log event 12
I20260501 13:58:32.482586 3943 tablet_replica.cc:333] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:32.483332 3943 raft_consensus.cc:2243] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:32.483884 3943 raft_consensus.cc:2272] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:32.473588 4231 tablet_bootstrap.cc:492] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:32.487210 4231 ts_tablet_manager.cc:1403] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.000s
I20260501 13:58:32.489997 4231 raft_consensus.cc:359] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.493058 3221 catalog_manager.cc:5671] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:32.496610 3213 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 034a29eda5fe4b43ac6dcf3616299e5d (table default.table_e3c7989801224024a8debb2eda1686e8 [id=c8b517f0258c4d35a40759e478d314ee]) successfully deleted
I20260501 13:58:32.501837 3227 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Sending DeleteTablet for 3 replicas of tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:32.502614 3943 ts_tablet_manager.cc:1916] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:32.505761 3227 catalog_manager.cc:6741] ReplaceTablet: tablet 19f08ae176df4a369c635a562b3dc3ba deleted and replaced by tablet 7b9f3b4229594e13959fec0aa8281c03
I20260501 13:58:32.504554 4021 tablet_bootstrap.cc:492] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:32.507167 4021 ts_tablet_manager.cc:1403] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.116s user 0.009s sys 0.000s
I20260501 13:58:32.509819 4021 raft_consensus.cc:359] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.510439 3651 tablet_service.cc:1558] Processing DeleteTablet for tablet 19f08ae176df4a369c635a562b3dc3ba with delete_type TABLET_DATA_DELETED (replaced by tablet 7b9f3b4229594e13959fec0aa8281c03) from {username='slave'} at 127.0.0.1:49092
I20260501 13:58:32.504444 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet 19f08ae176df4a369c635a562b3dc3ba with delete_type TABLET_DATA_DELETED (replaced by tablet 7b9f3b4229594e13959fec0aa8281c03) from {username='slave'} at 127.0.0.1:58038
W20260501 13:58:32.516355 3214 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 19f08ae176df4a369c635a562b3dc3ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19f08ae176df4a369c635a562b3dc3ba already in progress: creating tablet
I20260501 13:58:32.529306 4231 raft_consensus.cc:385] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.529641 4231 raft_consensus.cc:740] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.532120 4231 consensus_queue.cc:260] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.515702 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 19f08ae176df4a369c635a562b3dc3ba with delete_type TABLET_DATA_DELETED (replaced by tablet 7b9f3b4229594e13959fec0aa8281c03) from {username='slave'} at 127.0.0.1:46676
I20260501 13:58:32.533362 4021 raft_consensus.cc:385] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.533664 4021 raft_consensus.cc:740] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.534258 4021 consensus_queue.cc:260] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.535151 3227 master_service.cc:954] ReplaceTablet: received request to replace tablet ba69864093504babbb00fb6170774778 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:32.537164 3936 tablet_bootstrap.cc:654] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:32.548631 3227 catalog_manager.cc:5504] Ignoring report from unknown tablet a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:32.552644 4231 ts_tablet_manager.cc:1434] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.065s user 0.006s sys 0.001s
I20260501 13:58:32.554994 4021 ts_tablet_manager.cc:1434] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.047s user 0.005s sys 0.000s
I20260501 13:58:32.555891 4021 tablet_bootstrap.cc:492] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:32.562094 4231 tablet_bootstrap.cc:492] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:32.578455 3387 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet c8ed945f80854c06ae817814f5070b13
I20260501 13:58:32.580387 3226 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50472 (request call id 66) took 5720 ms. Trace:
I20260501 13:58:32.580773 3226 rpcz_store.cc:276] 0501 13:58:26.860426 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:26.860554 (+ 128us) service_pool.cc:224] Handling call
0501 13:58:31.895248 (+5034694us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:32.580307 (+685059us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":7439,"Hive Metastore.run_cpu_time_us":315,"Hive Metastore.run_wall_time_us":531770}
I20260501 13:58:32.592578 4245 raft_consensus.cc:493] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:32.593132 4245 raft_consensus.cc:515] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:32.596961 3226 catalog_manager.cc:5504] Ignoring report from unknown tablet a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:32.598596 4245 leader_election.cc:290] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:32.599956 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19f08ae176df4a369c635a562b3dc3ba" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:32.600106 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19f08ae176df4a369c635a562b3dc3ba" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:32.601488 4231 tablet_bootstrap.cc:654] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:32.601444 3608 leader_election.cc:343] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:58:32.602073 3609 leader_election.cc:343] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:32.602515 3609 leader_election.cc:304] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:32.603546 4245 raft_consensus.cc:2749] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:32.604707 4021 tablet_bootstrap.cc:654] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.615015 3936 tablet_bootstrap.cc:492] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:32.615435 3936 ts_tablet_manager.cc:1403] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.137s user 0.005s sys 0.004s
W20260501 13:58:32.617194 3213 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.190:38561, user_credentials={real_user=slave}} blocked reactor thread for 44919.6us
I20260501 13:58:32.617764 3936 raft_consensus.cc:359] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.618358 3936 raft_consensus.cc:385] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.618625 3936 raft_consensus.cc:740] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.619203 3936 consensus_queue.cc:260] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.621362 3936 ts_tablet_manager.cc:1434] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260501 13:58:32.641877 4231 tablet_bootstrap.cc:492] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:32.647050 3943 ts_tablet_manager.cc:1929] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:32.647440 3943 log.cc:1199] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/ba69864093504babbb00fb6170774778
I20260501 13:58:32.648783 3936 tablet_bootstrap.cc:492] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:32.650722 4231 ts_tablet_manager.cc:1403] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.001s
I20260501 13:58:32.653445 4231 raft_consensus.cc:359] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.654119 4231 raft_consensus.cc:385] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.658636 3943 ts_tablet_manager.cc:1950] T ba69864093504babbb00fb6170774778 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:32.663053 3943 tablet_replica.cc:333] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:32.663736 3943 raft_consensus.cc:2243] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:32.664377 3943 pending_rounds.cc:70] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: Trying to abort 1 pending ops.
I20260501 13:58:32.664592 3943 pending_rounds.cc:77] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229465356210176 op_type: NO_OP noop_request { }
I20260501 13:58:32.665006 3943 raft_consensus.cc:2889] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:32.665215 4021 tablet_bootstrap.cc:492] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:32.665351 3943 raft_consensus.cc:2272] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:32.665583 4021 ts_tablet_manager.cc:1403] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.110s user 0.005s sys 0.003s
I20260501 13:58:32.668067 4021 raft_consensus.cc:359] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.668591 3936 tablet_bootstrap.cc:654] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.668669 4021 raft_consensus.cc:385] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.668918 4021 raft_consensus.cc:740] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.654381 4231 raft_consensus.cc:740] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.676524 4245 raft_consensus.cc:493] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:32.677040 4245 raft_consensus.cc:515] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.678905 4245 leader_election.cc:290] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:32.679963 3943 ts_tablet_manager.cc:1916] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:32.680929 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a626cb9dab0463eb9b23098a8adfd7d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:32.681869 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a626cb9dab0463eb9b23098a8adfd7d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:32.681912 4231 consensus_queue.cc:260] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.682137 4021 consensus_queue.cc:260] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.682955 3515 raft_consensus.cc:2468] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:32.685926 4231 ts_tablet_manager.cc:1434] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20260501 13:58:32.686378 3609 leader_election.cc:304] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:32.687428 4245 raft_consensus.cc:2804] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:32.687794 4245 raft_consensus.cc:493] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:32.688091 4245 raft_consensus.cc:3060] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.688167 4231 tablet_bootstrap.cc:492] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
W20260501 13:58:32.689862 3608 leader_election.cc:343] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:32.702351 3221 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50472:
table { table_name: "default.Table_ed51787806aa40069d50f787cc85ae8f" } new_table_name: "default.Table_e7ee55876f2f41cfa624a58f23f9d415" modify_external_catalogs: true
I20260501 13:58:32.705050 3943 ts_tablet_manager.cc:1929] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:32.705428 3943 log.cc:1199] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:32.706127 3943 ts_tablet_manager.cc:1950] T a9ecb5b51ae04f32a4f07f828d563fa4 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:32.710297 4245 raft_consensus.cc:515] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.711225 4231 tablet_bootstrap.cc:654] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.713008 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a626cb9dab0463eb9b23098a8adfd7d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:32.713544 3515 raft_consensus.cc:3060] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.713865 4021 ts_tablet_manager.cc:1434] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20260501 13:58:32.716199 4021 tablet_bootstrap.cc:492] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:32.718801 3943 tablet_replica.cc:333] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:32.719440 3943 raft_consensus.cc:2243] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:32.719893 3943 raft_consensus.cc:2272] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:32.720523 3515 raft_consensus.cc:2468] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:32.721306 3223 master_service.cc:954] ReplaceTablet: received request to replace tablet ba69864093504babbb00fb6170774778 from {username='slave'} at 127.0.0.1:50472
I20260501 13:58:32.721791 4021 tablet_bootstrap.cc:654] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.721760 3609 leader_election.cc:304] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:32.722659 4163 raft_consensus.cc:2804] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:32.723110 4163 raft_consensus.cc:697] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:32.723869 4163 consensus_queue.cc:237] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.724433 4245 leader_election.cc:290] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:32.725692 3943 ts_tablet_manager.cc:1916] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:32.732191 3936 tablet_bootstrap.cc:492] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:32.737381 3493 tablet_service.cc:1511] Processing CreateTablet for tablet 7b9f3b4229594e13959fec0aa8281c03 (DEFAULT_TABLE table=default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:32.738535 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b9f3b4229594e13959fec0aa8281c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:32.739032 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a626cb9dab0463eb9b23098a8adfd7d" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
W20260501 13:58:32.741156 3608 leader_election.cc:343] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:32.746656 3936 ts_tablet_manager.cc:1403] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.098s user 0.009s sys 0.005s
I20260501 13:58:32.751426 3936 raft_consensus.cc:359] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.752204 3936 raft_consensus.cc:385] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.752512 3936 raft_consensus.cc:740] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.753952 3936 consensus_queue.cc:260] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.757793 1090 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu with pid 3190
I20260501 13:58:32.759804 4021 tablet_bootstrap.cc:492] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:32.760214 4021 ts_tablet_manager.cc:1403] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.002s
I20260501 13:58:32.765471 3936 ts_tablet_manager.cc:1434] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260501 13:58:32.776803 4231 tablet_bootstrap.cc:492] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:32.777802 3943 ts_tablet_manager.cc:1929] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:32.778190 3943 log.cc:1199] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:58:32.778538 4231 ts_tablet_manager.cc:1403] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.002s
I20260501 13:58:32.778875 3943 ts_tablet_manager.cc:1950] T 6309ec735c424460ad5a74c6b7ec0d8a P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:32.762611 4021 raft_consensus.cc:359] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.779381 4021 raft_consensus.cc:385] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.779657 4021 raft_consensus.cc:740] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.779879 4163 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:32.780315 4163 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.780346 4021 consensus_queue.cc:260] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.782212 4163 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:32.785765 4021 ts_tablet_manager.cc:1434] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20260501 13:58:32.786645 4021 tablet_bootstrap.cc:492] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
W20260501 13:58:32.786983 3608 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:58:32.787241 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba69864093504babbb00fb6170774778" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:32.787902 3515 raft_consensus.cc:2468] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:32.791851 3609 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2
I20260501 13:58:32.792081 4231 raft_consensus.cc:359] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.792765 4231 raft_consensus.cc:385] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.793051 4231 raft_consensus.cc:740] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.793691 4231 consensus_queue.cc:260] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.795782 4231 ts_tablet_manager.cc:1434] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.017s user 0.000s sys 0.005s
I20260501 13:58:32.797364 4163 raft_consensus.cc:2804] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:32.797708 4163 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:32.798030 4163 raft_consensus.cc:3060] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.798524 4021 tablet_bootstrap.cc:654] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.811951 4163 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.823843 4021 tablet_bootstrap.cc:492] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:32.824283 4021 ts_tablet_manager.cc:1403] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.000s
I20260501 13:58:32.826790 4021 raft_consensus.cc:359] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.827517 4021 raft_consensus.cc:385] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.827672 4231 tablet_bootstrap.cc:492] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:32.827798 4021 raft_consensus.cc:740] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.828490 4021 consensus_queue.cc:260] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.836050 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba69864093504babbb00fb6170774778" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:32.836642 3515 raft_consensus.cc:3060] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.839614 4021 ts_tablet_manager.cc:1434] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260501 13:58:32.840488 4021 tablet_bootstrap.cc:492] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:32.845538 4231 tablet_bootstrap.cc:654] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.849642 4021 tablet_bootstrap.cc:654] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.859092 3942 ts_tablet_manager.cc:1929] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:32.849464 3515 raft_consensus.cc:2468] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
W20260501 13:58:32.861620 3608 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:58:32.867034 3609 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2
I20260501 13:58:32.867890 4245 raft_consensus.cc:2804] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:32.868330 4245 raft_consensus.cc:697] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:32.869112 4245 consensus_queue.cc:237] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.875192 4021 tablet_bootstrap.cc:492] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:32.875617 4021 ts_tablet_manager.cc:1403] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.006s
I20260501 13:58:32.864310 4163 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:32.877925 3942 log.cc:1199] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/034a29eda5fe4b43ac6dcf3616299e5d
I20260501 13:58:32.878202 4021 raft_consensus.cc:359] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.878839 4021 raft_consensus.cc:385] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.879104 4021 raft_consensus.cc:740] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.879905 4021 consensus_queue.cc:260] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.880899 3942 ts_tablet_manager.cc:1950] T 034a29eda5fe4b43ac6dcf3616299e5d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:32.889051 4021 ts_tablet_manager.cc:1434] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260501 13:58:32.893121 3942 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49092 (request call id 78) took 1194 ms. Trace:
I20260501 13:58:32.902887 4231 tablet_bootstrap.cc:492] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:32.903364 4231 ts_tablet_manager.cc:1403] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.002s
I20260501 13:58:32.909060 4231 raft_consensus.cc:359] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.909904 4231 raft_consensus.cc:385] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:32.910300 4231 raft_consensus.cc:740] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:32.911068 4231 consensus_queue.cc:260] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.913975 3942 rpcz_store.cc:276] 0501 13:58:31.698303 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:31.698425 (+ 122us) service_pool.cc:224] Handling call
0501 13:58:32.355140 (+656715us) ts_tablet_manager.cc:1151] Deleting tablet 034a29eda5fe4b43ac6dcf3616299e5d
0501 13:58:32.355182 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:58:32.622777 (+267595us) tablet_metadata.cc:665] Metadata flushed
0501 13:58:32.858940 (+236163us) tablet_metadata.cc:665] Metadata flushed
0501 13:58:32.893048 (+ 34108us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":653121}
I20260501 13:58:32.920939 4231 ts_tablet_manager.cc:1434] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260501 13:58:32.921487 3942 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49092 (request call id 79) took 1221 ms. Trace:
I20260501 13:58:32.921905 3942 rpcz_store.cc:276] 0501 13:58:31.700224 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:31.700338 (+ 114us) service_pool.cc:224] Handling call
0501 13:58:32.921086 (+1220748us) ts_tablet_manager.cc:1151] Deleting tablet 034a29eda5fe4b43ac6dcf3616299e5d
0501 13:58:32.921129 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:58:32.921418 (+ 289us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1218799}
I20260501 13:58:32.930303 4231 tablet_bootstrap.cc:492] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:32.936522 4243 raft_consensus.cc:493] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:32.923522 3942 tablet_replica.cc:333] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:32.937043 4243 raft_consensus.cc:515] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.938100 3942 raft_consensus.cc:2243] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:32.939018 4243 leader_election.cc:290] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:32.940479 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545ced72d1cd47faa827d7b00df372b7" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:32.938867 3942 pending_rounds.cc:70] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: Trying to abort 1 pending ops.
I20260501 13:58:32.941033 3942 pending_rounds.cc:77] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229467127271424 op_type: NO_OP noop_request { }
I20260501 13:58:32.941108 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545ced72d1cd47faa827d7b00df372b7" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:32.941370 3942 raft_consensus.cc:2889] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:32.941655 3673 raft_consensus.cc:2468] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
W20260501 13:58:32.941994 3757 leader_election.cc:343] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:32.941673 3942 raft_consensus.cc:2272] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:32.940479 4231 tablet_bootstrap.cc:654] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:32.942742 3756 leader_election.cc:304] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:32.944532 4243 raft_consensus.cc:2804] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:32.944881 4243 raft_consensus.cc:493] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:32.945194 4243 raft_consensus.cc:3060] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.952953 3942 ts_tablet_manager.cc:1916] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:32.953354 4243 raft_consensus.cc:515] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:32.956938 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545ced72d1cd47faa827d7b00df372b7" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:32.957106 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545ced72d1cd47faa827d7b00df372b7" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:32.957675 3673 raft_consensus.cc:3060] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.958339 4240 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:32.958828 4240 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.960668 4240 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:32.961476 3757 leader_election.cc:343] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:32.963198 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9ecb5b51ae04f32a4f07f828d563fa4" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:32.964032 3674 raft_consensus.cc:2393] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9154033cea374a10a5225c53b8bcf122 in current term 1: Already voted for candidate 5c239ccbdc554619a237d78e958ffbc2 in this term.
W20260501 13:58:32.964092 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:32.964872 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:32.965962 4240 raft_consensus.cc:3060] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:32.968734 3673 raft_consensus.cc:2468] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:32.969849 3756 leader_election.cc:304] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:32.972945 4240 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:32.977034 3942 ts_tablet_manager.cc:1929] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:32.977401 3942 log.cc:1199] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/ba69864093504babbb00fb6170774778
I20260501 13:58:32.978044 3942 ts_tablet_manager.cc:1950] T ba69864093504babbb00fb6170774778 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:32.979125 4243 leader_election.cc:290] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:32.979987 3942 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49092 (request call id 83) took 1164 ms. Trace:
I20260501 13:58:32.980279 3942 rpcz_store.cc:276] 0501 13:58:31.816058 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:31.816174 (+ 116us) service_pool.cc:224] Handling call
0501 13:58:32.923383 (+1107209us) ts_tablet_manager.cc:1151] Deleting tablet ba69864093504babbb00fb6170774778
0501 13:58:32.923421 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:58:32.960062 (+ 36641us) tablet_metadata.cc:665] Metadata flushed
0501 13:58:32.976905 (+ 16843us) tablet_metadata.cc:665] Metadata flushed
0501 13:58:32.979915 (+ 3010us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106673}
I20260501 13:58:32.981302 3942 tablet_replica.cc:333] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:32.981225 4245 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:32.981969 3942 raft_consensus.cc:2243] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:32.981644 4245 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.983790 4245 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:32.985448 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6309ec735c424460ad5a74c6b7ec0d8a" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:32.986119 3515 raft_consensus.cc:2468] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:32.982378 3942 raft_consensus.cc:2272] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:32.987473 3609 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:32.988924 3942 ts_tablet_manager.cc:1916] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:58:32.990433 3608 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:58:32.992183 4245 raft_consensus.cc:2804] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:32.992626 4245 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:32.993029 4245 raft_consensus.cc:3060] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:33.001063 4245 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:32.980844 4243 raft_consensus.cc:2804] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:33.007157 4243 raft_consensus.cc:697] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:33.011900 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6309ec735c424460ad5a74c6b7ec0d8a" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:33.012478 3515 raft_consensus.cc:3060] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:33.020185 3515 raft_consensus.cc:2468] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:33.020344 3942 ts_tablet_manager.cc:1929] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:33.020707 3942 log.cc:1199] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:33.021319 3942 ts_tablet_manager.cc:1950] T a9ecb5b51ae04f32a4f07f828d563fa4 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:33.022648 3609 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:33.023195 3942 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49092 (request call id 88) took 1012 ms. Trace:
I20260501 13:58:33.023463 4163 raft_consensus.cc:2804] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:33.023897 4163 raft_consensus.cc:697] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:33.024621 4163 consensus_queue.cc:237] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:33.023480 3942 rpcz_store.cc:276] 0501 13:58:32.010862 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:32.011240 (+ 378us) service_pool.cc:224] Handling call
0501 13:58:32.981160 (+969920us) ts_tablet_manager.cc:1151] Deleting tablet a9ecb5b51ae04f32a4f07f828d563fa4
0501 13:58:32.981197 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0501 13:58:33.004857 (+ 23660us) tablet_metadata.cc:665] Metadata flushed
0501 13:58:33.020212 (+ 15355us) tablet_metadata.cc:665] Metadata flushed
0501 13:58:33.023127 (+ 2915us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":954944}
I20260501 13:58:33.029683 3942 tablet_replica.cc:333] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:33.041951 3942 raft_consensus.cc:2243] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:33.043242 3942 pending_rounds.cc:70] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: Trying to abort 1 pending ops.
I20260501 13:58:33.043488 3942 pending_rounds.cc:77] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229467762757632 op_type: NO_OP noop_request { }
I20260501 13:58:33.043803 3942 raft_consensus.cc:2889] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:33.008634 4243 consensus_queue.cc:237] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:33.044144 3942 raft_consensus.cc:2272] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:58:33.054643 3608 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:58:33.056458 4267 consensus_queue.cc:1175] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20260501 13:58:33.057050 4245 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:33.060343 3756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.130:45857, user_credentials={real_user=slave}} blocked reactor thread for 90590.4us
I20260501 13:58:33.062584 3942 ts_tablet_manager.cc:1916] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:33.076881 4231 tablet_bootstrap.cc:492] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:33.077353 4231 ts_tablet_manager.cc:1403] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.147s user 0.023s sys 0.028s
I20260501 13:58:33.080093 4231 raft_consensus.cc:359] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:33.080792 4231 raft_consensus.cc:385] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:33.081059 4231 raft_consensus.cc:740] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:33.086781 4231 consensus_queue.cc:260] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:33.088169 3942 ts_tablet_manager.cc:1929] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:33.088517 3942 log.cc:1199] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:58:33.089190 3942 ts_tablet_manager.cc:1950] T 6309ec735c424460ad5a74c6b7ec0d8a P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:33.091856 3942 tablet_replica.cc:333] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:33.092474 3942 raft_consensus.cc:2243] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:33.092897 3942 raft_consensus.cc:2272] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:33.094506 3942 ts_tablet_manager.cc:1916] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:33.101761 4231 ts_tablet_manager.cc:1434] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.024s user 0.006s sys 0.001s
W20260501 13:58:33.109056 3758 connection.cc:570] server connection from 127.0.0.1:46676 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:58:33.110601 3879 connection.cc:570] client connection to 127.1.16.189:44255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:58:33.112710 3879 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.16.189:44255)
W20260501 13:58:33.113521 3879 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.16.189:44255)
W20260501 13:58:33.114192 3879 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.16.189:44255)
W20260501 13:58:33.114974 3879 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.16.189:44255)
I20260501 13:58:33.116324 4231 tablet_bootstrap.cc:492] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
W20260501 13:58:33.120492 3757 connection.cc:570] client connection to 127.1.16.189:44255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 13:58:33.123137 3942 ts_tablet_manager.cc:1929] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:33.123499 3942 log.cc:1199] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:33.124140 3942 ts_tablet_manager.cc:1950] T 19f08ae176df4a369c635a562b3dc3ba P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:33.122305 4231 tablet_bootstrap.cc:654] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:33.131772 3609 connection.cc:570] client connection to 127.1.16.189:44255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:58:33.145417 3608 connection.cc:570] server connection from 127.0.0.1:49092 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:58:33.146309 3449 connection.cc:570] client connection to 127.1.16.189:44255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 13:58:33.151748 4245 raft_consensus.cc:493] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:33.152221 4245 raft_consensus.cc:515] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:33.154093 4245 leader_election.cc:290] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:33.177263 3450 connection.cc:570] server connection from 127.0.0.1:58038 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:58:33.201824 3138 connection.cc:570] server connection from 127.0.0.1:54904 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 13:58:33.204440 4274 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 13:58:33.133827 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.1.16.189:44255 (local address 127.1.16.130:34203)
0501 13:58:33.203361 (+ 69534us) negotiation.cc:107] Waiting for socket to connect
0501 13:58:33.203616 (+ 255us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":68779,"threads_started":1}
I20260501 13:58:33.207118 4276 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 15dd70d7c5494c3c911f0a780dc9a2af)
W20260501 13:58:33.207897 4269 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 13:58:33.119867 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.1.16.189:44255 (local address 127.0.0.1:37482)
0501 13:58:33.199964 (+ 80097us) negotiation.cc:107] Waiting for socket to connect
0501 13:58:33.200230 (+ 266us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":79401,"threads_started":1}
I20260501 13:58:33.207737 4276 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:33.205829 4277 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 15dd70d7c5494c3c911f0a780dc9a2af)
I20260501 13:58:33.208755 4277 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:33.212854 3163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 6 candidate_status { last_received { term: 5 index: 111 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:33.213418 3163 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b0f8911dd5d431d9d8d3124a4456262 in term 5.
I20260501 13:58:33.214838 4036 leader_election.cc:304] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262; no voters:
I20260501 13:58:33.232120 4277 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:33.236311 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d129cc8c1fc840cd9e00d4ad22ff236c" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:33.237044 3515 raft_consensus.cc:2468] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:33.242259 3609 leader_election.cc:304] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:33.243999 4245 raft_consensus.cc:2804] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:33.244346 4245 raft_consensus.cc:493] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:33.244668 4245 raft_consensus.cc:3060] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:33.248943 4240 raft_consensus.cc:493] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:33.249423 4240 raft_consensus.cc:515] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:33.251362 4240 leader_election.cc:290] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:33.253631 3448 leader_election.cc:343] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:33.253953 4245 raft_consensus.cc:515] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:33.258818 4245 leader_election.cc:290] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:33.264433 4275 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 13:58:33.148227 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.1.16.189:44255 (local address 127.1.16.129:34439)
0501 13:58:33.206322 (+ 58095us) negotiation.cc:107] Waiting for socket to connect
0501 13:58:33.263207 (+ 56885us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":57286,"thread_start_us":83360,"threads_started":1}
I20260501 13:58:33.268424 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19f08ae176df4a369c635a562b3dc3ba" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:33.269049 3821 raft_consensus.cc:2468] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:33.270586 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d129cc8c1fc840cd9e00d4ad22ff236c" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:33.270989 4276 leader_election.cc:290] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255)
I20260501 13:58:33.271219 3821 raft_consensus.cc:2468] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:33.271512 4276 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260501 13:58:33.272957 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d129cc8c1fc840cd9e00d4ad22ff236c" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:33.273526 3821 raft_consensus.cc:3060] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:33.271802 4276 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 15dd70d7c5494c3c911f0a780dc9a2af)
I20260501 13:58:33.274317 4276 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 5 FOLLOWER]: Advancing to term 6
I20260501 13:58:33.280377 4231 tablet_bootstrap.cc:492] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:33.280864 4231 ts_tablet_manager.cc:1403] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.165s user 0.044s sys 0.057s
I20260501 13:58:33.282534 3821 raft_consensus.cc:2468] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:33.284758 4231 raft_consensus.cc:359] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:33.285388 4231 raft_consensus.cc:385] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:33.285640 4231 raft_consensus.cc:740] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
W20260501 13:58:33.287456 3609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:44335, user_credentials={real_user=slave}} blocked reactor thread for 45296.7us
W20260501 13:58:33.288388 4037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
I20260501 13:58:33.288985 4231 consensus_queue.cc:260] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:33.289559 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d129cc8c1fc840cd9e00d4ad22ff236c" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:33.290154 3515 raft_consensus.cc:3060] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:33.300009 4231 ts_tablet_manager.cc:1434] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20260501 13:58:33.299871 3515 raft_consensus.cc:2468] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
W20260501 13:58:33.302704 4272 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 13:58:33.122236 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.1.16.189:44255 (local address 127.1.16.131:49971)
0501 13:58:33.301305 (+179069us) negotiation.cc:107] Waiting for socket to connect
0501 13:58:33.301569 (+ 264us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":143758,"thread_start_us":144856,"threads_started":1}
W20260501 13:58:33.302709 4037 leader_election.cc:336] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255): Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
I20260501 13:58:33.308146 3608 leader_election.cc:304] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:33.308787 3448 leader_election.cc:304] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:33.309767 4245 raft_consensus.cc:2804] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:33.309749 4240 raft_consensus.cc:2804] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:33.310111 4240 raft_consensus.cc:493] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:33.310436 4240 raft_consensus.cc:3060] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:33.310914 4245 raft_consensus.cc:697] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:33.311712 4245 consensus_queue.cc:237] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:33.321552 3140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20260501 13:58:33.321722 4240 raft_consensus.cc:515] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:33.324965 3448 leader_election.cc:343] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:33.325457 4240 leader_election.cc:290] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:33.325976 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19f08ae176df4a369c635a562b3dc3ba" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:33.326786 3821 raft_consensus.cc:3060] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:58:33.326706 4036 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.190:38561, user_credentials={real_user=slave}} blocked reactor thread for 112074us
I20260501 13:58:33.328733 4231 tablet_bootstrap.cc:492] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
W20260501 13:58:33.334111 3140 leader_election.cc:336] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255): Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
I20260501 13:58:33.338227 4231 tablet_bootstrap.cc:654] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:33.346143 4163 consensus_queue.cc:1048] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:33.347551 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 6 candidate_status { last_received { term: 5 index: 111 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:33.351851 3821 raft_consensus.cc:2468] T 19f08ae176df4a369c635a562b3dc3ba P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:33.353147 3448 leader_election.cc:304] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:33.354202 4240 raft_consensus.cc:2804] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:33.375790 4276 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:33.364666 4240 raft_consensus.cc:697] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:33.378479 3163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b0f8911dd5d431d9d8d3124a4456262" candidate_term: 6 candidate_status { last_received { term: 5 index: 111 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016"
I20260501 13:58:33.379038 3163 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 5 FOLLOWER]: Advancing to term 6
I20260501 13:58:33.379369 4080 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a1d2cd221924677a3ef81b081e5e016 in current term 6: Already voted for candidate 6b0f8911dd5d431d9d8d3124a4456262 in this term.
I20260501 13:58:33.384220 3139 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016; no voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:33.400564 3163 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b0f8911dd5d431d9d8d3124a4456262 in term 6.
I20260501 13:58:33.404899 4231 tablet_bootstrap.cc:492] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:33.405444 4231 ts_tablet_manager.cc:1403] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.077s user 0.007s sys 0.018s
I20260501 13:58:33.406320 4163 consensus_queue.cc:1048] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:33.407828 4277 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260501 13:58:33.411707 4036 leader_election.cc:304] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262; no voters:
I20260501 13:58:33.412631 4278 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 6 FOLLOWER]: Leader election won for term 6
I20260501 13:58:33.419631 4240 consensus_queue.cc:237] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:33.411744 4276 leader_election.cc:290] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 6 election: Requested vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255)
W20260501 13:58:33.421305 4037 leader_election.cc:336] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255): Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111)
I20260501 13:58:33.432030 4278 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [term 6 LEADER]: Becoming Leader. State: Replica: 6b0f8911dd5d431d9d8d3124a4456262, State: Running, Role: LEADER
I20260501 13:58:33.433189 4278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 5.111, Last appended by leader: 111, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:33.407855 4231 raft_consensus.cc:359] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:33.436614 4231 raft_consensus.cc:385] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:33.436918 4231 raft_consensus.cc:740] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:33.437546 4231 consensus_queue.cc:260] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:33.439751 4231 ts_tablet_manager.cc:1434] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.034s user 0.004s sys 0.000s
I20260501 13:58:33.442708 4276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0f8911dd5d431d9d8d3124a4456262. Latest consensus state: current_term: 6 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:33.443569 4276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: LEADER
I20260501 13:58:33.679845 4243 raft_consensus.cc:493] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:33.680393 4243 raft_consensus.cc:515] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:33.682330 4243 leader_election.cc:290] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:33.683197 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a626cb9dab0463eb9b23098a8adfd7d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:33.683512 3673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a626cb9dab0463eb9b23098a8adfd7d" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:33.683828 3515 raft_consensus.cc:2393] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c239ccbdc554619a237d78e958ffbc2 in current term 1: Already voted for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in this term.
I20260501 13:58:33.684885 3757 leader_election.cc:304] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:33.685510 4243 raft_consensus.cc:3060] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:33.692305 4243 raft_consensus.cc:2749] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:33.761656 4163 consensus_queue.cc:1048] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:33.784842 4163 consensus_queue.cc:1048] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:58:33.849769 4037 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 -> Peer 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255): Couldn't send request to peer 15dd70d7c5494c3c911f0a780dc9a2af. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260501 13:58:33.909045 4243 consensus_queue.cc:1048] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:33.915364 4240 raft_consensus.cc:493] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:33.915863 4240 raft_consensus.cc:515] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } }
I20260501 13:58:33.919121 4240 leader_election.cc:290] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:33.920980 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545ced72d1cd47faa827d7b00df372b7" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:33.922165 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "545ced72d1cd47faa827d7b00df372b7" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:33.923436 3448 leader_election.cc:304] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:33.924503 4240 raft_consensus.cc:2749] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:33.930143 3515 raft_consensus.cc:3060] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:33.954365 4302 consensus_queue.cc:1048] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:58:33.982204 3163 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 6 FOLLOWER]: Refusing update from remote peer 6b0f8911dd5d431d9d8d3124a4456262: Log matching property violated. Preceding OpId in replica: term: 5 index: 111. Preceding OpId from leader: term: 6 index: 112. (index mismatch)
I20260501 13:58:33.983654 4276 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 110, Time since last communication: 0.000s
I20260501 13:58:33.989068 4304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0f8911dd5d431d9d8d3124a4456262. Latest consensus state: current_term: 6 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:33.989646 4304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:33.990334 1090 external_mini_cluster.cc:1366] Running /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
/tmp/dist-test-task2LI5eU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.189:44255
--webserver_interface=127.1.16.189
--webserver_port=36257
--builtin_ntp_servers=127.1.16.148:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--hive_metastore_uris=thrift://127.0.0.1:45913
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260501 13:58:34.018199 4304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:34.018975 4304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:34.039162 4276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "6b0f8911dd5d431d9d8d3124a4456262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:34.039881 4276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 [sys.catalog]: This master's current role is: LEADER
I20260501 13:58:34.110121 4163 consensus_queue.cc:1048] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:34.119928 4295 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 13:58:34.139688 4245 consensus_queue.cc:1048] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:34.187879 4240 raft_consensus.cc:493] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:34.188086 4295 catalog_manager.cc:679] Loaded metadata for table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]
I20260501 13:58:34.188374 4240 raft_consensus.cc:515] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:34.190255 4240 leader_election.cc:290] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:34.194705 3448 leader_election.cc:343] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 7b9f3b4229594e13959fec0aa8281c03
W20260501 13:58:34.195791 3448 leader_election.cc:343] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 7b9f3b4229594e13959fec0aa8281c03
I20260501 13:58:34.196527 4295 catalog_manager.cc:679] Loaded metadata for table default.table_1f445634a0fc4676a2e7bf208b0ea1de [id=18ac75baaaf7485da21228e06a780ad2]
I20260501 13:58:34.196604 3448 leader_election.cc:304] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:34.198237 4295 catalog_manager.cc:679] Loaded metadata for table default.table_ed51787806aa40069d50f787cc85ae8f [id=8d51c67732064782bd5d53da7178ba87]
W20260501 13:58:34.200464 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:58:34.202641 4314 raft_consensus.cc:2749] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:34.204164 4295 catalog_manager.cc:679] Loaded metadata for table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]
I20260501 13:58:34.209998 4295 catalog_manager.cc:679] Loaded metadata for table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef]
I20260501 13:58:34.227523 4295 tablet_loader.cc:96] loaded metadata for tablet 258801137bf64cdd8588cac0d48c5c7d (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee])
I20260501 13:58:34.228910 4295 tablet_loader.cc:96] loaded metadata for tablet 545ced72d1cd47faa827d7b00df372b7 (table default.table_1f445634a0fc4676a2e7bf208b0ea1de [id=18ac75baaaf7485da21228e06a780ad2])
I20260501 13:58:34.231141 4295 tablet_loader.cc:96] loaded metadata for tablet 7a626cb9dab0463eb9b23098a8adfd7d (table default.table_ed51787806aa40069d50f787cc85ae8f [id=8d51c67732064782bd5d53da7178ba87])
I20260501 13:58:34.232494 4295 tablet_loader.cc:96] loaded metadata for tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_1d615bc9ac7a4db39b4d4afb4ef38d8a [id=b1334bc9986e4463add9b290ab6294ef])
I20260501 13:58:34.234540 4295 tablet_loader.cc:96] loaded metadata for tablet d129cc8c1fc840cd9e00d4ad22ff236c (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129])
I20260501 13:58:34.238194 4295 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 13:58:34.250555 4295 catalog_manager.cc:1269] Loaded cluster ID: 4281e294e62c41c781e2041e1b24de7c
I20260501 13:58:34.250900 4295 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 13:58:34.263268 4295 catalog_manager.cc:1514] Loading token signing keys...
I20260501 13:58:34.274585 4295 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Loaded TSK: 0
I20260501 13:58:34.276013 4295 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 13:58:34.287496 4295 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260501 13:58:34.299818 4295 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: 12
I20260501 13:58:34.303478 4314 consensus_queue.cc:1048] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:34.442114 4055 master_service.cc:954] ReplaceTablet: received request to replace tablet 258801137bf64cdd8588cac0d48c5c7d from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:34.453209 4064 master_service.cc:954] ReplaceTablet: received request to replace tablet a9ecb5b51ae04f32a4f07f828d563fa4 from {username='slave'} at 127.0.0.1:35438
W20260501 13:58:34.457962 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:34.467393 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:34.506554 3867 heartbeater.cc:499] Master 127.1.16.188:38253 was elected leader, sending a full tablet report...
W20260501 13:58:34.523694 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:34.441325 4052 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_aafd3ab3f9f44a74af41aae60cfe40fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:34.451117 4066 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_f701ba5ffc3845bf8cb5e55102e60688"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:34.447824 4056 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_1d615bc9ac7a4db39b4d4afb4ef38d8a" } new_table_name: "default.Table_4db61a7562964779afbc5ff93b00a5f5" modify_external_catalogs: true
I20260501 13:58:34.436877 4050 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_5c8ebfa533ee4e9a94d8200053bfe4c0" } modify_external_catalogs: true
I20260501 13:58:34.448776 4059 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_f52f85ad11fb4023adead2ab9c5eb694"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:34.442376 4054 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_06e2e16b792549e68d2e01ce3e3ce8b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:34.535521 4050 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_5c8ebfa533ee4e9a94d8200053bfe4c0" } modify_external_catalogs: true
I20260501 13:58:34.449977 4063 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_ed51787806aa40069d50f787cc85ae8f" } new_table_name: "default.Table_e7ee55876f2f41cfa624a58f23f9d415" modify_external_catalogs: true
I20260501 13:58:34.374046 4061 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_2840ee60b88a494f9b40b448d78b0000" } modify_external_catalogs: true
I20260501 13:58:34.561040 4061 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_2840ee60b88a494f9b40b448d78b0000" } modify_external_catalogs: true
I20260501 13:58:34.582674 4055 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet 258801137bf64cdd8588cac0d48c5c7d
I20260501 13:58:34.581409 4064 catalog_manager.cc:5671] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } }
I20260501 13:58:34.593461 4064 catalog_manager.cc:5671] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } }
I20260501 13:58:34.601641 4055 catalog_manager.cc:6741] ReplaceTablet: tablet 258801137bf64cdd8588cac0d48c5c7d deleted and replaced by tablet 80a2f6fab87a4ad980c7816f34f41aff
I20260501 13:58:34.603797 4064 catalog_manager.cc:5671] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } }
I20260501 13:58:34.605122 4064 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet d129cc8c1fc840cd9e00d4ad22ff236c (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]). Expected version 1 got 0
I20260501 13:58:34.643471 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet 258801137bf64cdd8588cac0d48c5c7d with delete_type TABLET_DATA_DELETED (replaced by tablet 80a2f6fab87a4ad980c7816f34f41aff) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:34.657462 4329 tablet_replica.cc:333] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:34.659726 4329 raft_consensus.cc:2243] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260501 13:58:34.661494 4309 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 13:58:34.662290 4309 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260501 13:58:34.663300 4329 raft_consensus.cc:2272] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:58:34.669008 4309 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260501 13:58:34.669865 3651 tablet_service.cc:1558] Processing DeleteTablet for tablet 258801137bf64cdd8588cac0d48c5c7d with delete_type TABLET_DATA_DELETED (replaced by tablet 80a2f6fab87a4ad980c7816f34f41aff) from {username='slave'} at 127.0.0.1:48686
W20260501 13:58:34.672964 4187 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b9f3b4229594e13959fec0aa8281c03 consensus state has no leader
I20260501 13:58:34.678725 4329 ts_tablet_manager.cc:1916] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:58:34.691484 4052 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_aafd3ab3f9f44a74af41aae60cfe40fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:34.694437 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 258801137bf64cdd8588cac0d48c5c7d with delete_type TABLET_DATA_DELETED (replaced by tablet 80a2f6fab87a4ad980c7816f34f41aff) from {username='slave'} at 127.0.0.1:47972
W20260501 13:58:34.695525 4066 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f701ba5ffc3845bf8cb5e55102e60688 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:34.697690 4056 master_service.cc:954] ReplaceTablet: received request to replace tablet ba69864093504babbb00fb6170774778 from {username='slave'} at 127.0.0.1:35438
W20260501 13:58:34.699566 4054 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_06e2e16b792549e68d2e01ce3e3ce8b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:34.699661 4330 tablet_replica.cc:333] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:34.701042 4330 raft_consensus.cc:2243] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:34.702065 4330 raft_consensus.cc:2272] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:58:34.703785 4059 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f52f85ad11fb4023adead2ab9c5eb694 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:34.704020 4330 ts_tablet_manager.cc:1916] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:34.711756 4050 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_1f445634a0fc4676a2e7bf208b0ea1de" } new_table_name: "default.Table_04a1387d0f0343998d94c1bb5bbff699" modify_external_catalogs: true
I20260501 13:58:34.712074 4331 tablet_replica.cc:333] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:34.712859 4331 raft_consensus.cc:2243] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:34.713796 4056 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_b07edfe247554299af96b76fee9a5b4e" } modify_external_catalogs: true
I20260501 13:58:34.718040 4329 ts_tablet_manager.cc:1929] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:34.724656 4329 log.cc:1199] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/258801137bf64cdd8588cac0d48c5c7d
I20260501 13:58:34.728178 4329 ts_tablet_manager.cc:1950] T 258801137bf64cdd8588cac0d48c5c7d P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:34.732308 4331 raft_consensus.cc:2272] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:34.734380 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 258801137bf64cdd8588cac0d48c5c7d (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:34.734921 4331 ts_tablet_manager.cc:1916] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:34.737296 4330 ts_tablet_manager.cc:1929] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260501 13:58:34.737643 4038 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b9f3b4229594e13959fec0aa8281c03 consensus state has no leader
I20260501 13:58:34.737854 4330 log.cc:1199] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/258801137bf64cdd8588cac0d48c5c7d
I20260501 13:58:34.738806 4330 ts_tablet_manager.cc:1950] T 258801137bf64cdd8588cac0d48c5c7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:34.738746 4056 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_b07edfe247554299af96b76fee9a5b4e" } modify_external_catalogs: true
I20260501 13:58:34.742894 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 258801137bf64cdd8588cac0d48c5c7d (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:34.767079 3493 tablet_service.cc:1511] Processing CreateTablet for tablet 80a2f6fab87a4ad980c7816f34f41aff (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:34.768379 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a2f6fab87a4ad980c7816f34f41aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:34.770862 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 80a2f6fab87a4ad980c7816f34f41aff (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:34.770823 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 80a2f6fab87a4ad980c7816f34f41aff (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:34.771828 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a2f6fab87a4ad980c7816f34f41aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:34.771831 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a2f6fab87a4ad980c7816f34f41aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:34.795552 4331 ts_tablet_manager.cc:1929] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:34.796993 4334 tablet.cc:1724] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:34.804507 4331 log.cc:1199] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/258801137bf64cdd8588cac0d48c5c7d
I20260501 13:58:34.807713 4336 tablet.cc:1724] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:34.810194 3719 heartbeater.cc:499] Master 127.1.16.188:38253 was elected leader, sending a full tablet report...
I20260501 13:58:34.817701 4055 master_service.cc:954] ReplaceTablet: received request to replace tablet 80a2f6fab87a4ad980c7816f34f41aff from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:34.819006 4331 ts_tablet_manager.cc:1950] T 258801137bf64cdd8588cac0d48c5c7d P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:34.821821 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 258801137bf64cdd8588cac0d48c5c7d (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:34.822178 4335 tablet.cc:1724] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 13:58:34.829936 4309 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
I20260501 13:58:34.835263 4338 tablet_bootstrap.cc:492] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
W20260501 13:58:34.844210 4309 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260501 13:58:34.844661 4309 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 13:58:34.845060 4309 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 13:58:34.845419 4309 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 13:58:34.845793 4309 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 13:58:34.855866 4055 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet 80a2f6fab87a4ad980c7816f34f41aff
I20260501 13:58:34.857360 4055 catalog_manager.cc:6741] ReplaceTablet: tablet 80a2f6fab87a4ad980c7816f34f41aff deleted and replaced by tablet d44e6e83e21f4c9485a4df3e40f8f72b
I20260501 13:58:34.859278 3491 tablet_service.cc:1558] Processing DeleteTablet for tablet 80a2f6fab87a4ad980c7816f34f41aff with delete_type TABLET_DATA_DELETED (replaced by tablet d44e6e83e21f4c9485a4df3e40f8f72b) from {username='slave'} at 127.0.0.1:52150
W20260501 13:58:34.861260 4037 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet 80a2f6fab87a4ad980c7816f34f41aff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80a2f6fab87a4ad980c7816f34f41aff already in progress: creating tablet
I20260501 13:58:34.863108 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 80a2f6fab87a4ad980c7816f34f41aff with delete_type TABLET_DATA_DELETED (replaced by tablet d44e6e83e21f4c9485a4df3e40f8f72b) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:34.863199 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 80a2f6fab87a4ad980c7816f34f41aff with delete_type TABLET_DATA_DELETED (replaced by tablet d44e6e83e21f4c9485a4df3e40f8f72b) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:34.864559 3561 heartbeater.cc:499] Master 127.1.16.188:38253 was elected leader, sending a full tablet report...
I20260501 13:58:34.867063 4338 tablet_bootstrap.cc:654] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:34.877846 4036 catalog_manager.cc:4982] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet 80a2f6fab87a4ad980c7816f34f41aff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80a2f6fab87a4ad980c7816f34f41aff already in progress: creating tablet
W20260501 13:58:34.878713 4036 catalog_manager.cc:4982] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet 80a2f6fab87a4ad980c7816f34f41aff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80a2f6fab87a4ad980c7816f34f41aff already in progress: creating tablet
I20260501 13:58:34.880362 4337 tablet_bootstrap.cc:492] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:34.912322 4341 tablet_bootstrap.cc:492] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:34.933015 4338 tablet_bootstrap.cc:492] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:34.934592 4338 ts_tablet_manager.cc:1403] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.100s user 0.011s sys 0.002s
I20260501 13:58:34.945379 4338 raft_consensus.cc:359] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:34.947504 4338 raft_consensus.cc:385] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:34.949550 3493 tablet_service.cc:1511] Processing CreateTablet for tablet d44e6e83e21f4c9485a4df3e40f8f72b (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:34.951426 4341 tablet_bootstrap.cc:654] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:34.952632 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d44e6e83e21f4c9485a4df3e40f8f72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:34.960333 3802 tablet_service.cc:1511] Processing CreateTablet for tablet d44e6e83e21f4c9485a4df3e40f8f72b (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:34.961468 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d44e6e83e21f4c9485a4df3e40f8f72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:34.986578 4338 raft_consensus.cc:740] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:34.951706 3654 tablet_service.cc:1511] Processing CreateTablet for tablet d44e6e83e21f4c9485a4df3e40f8f72b (DEFAULT_TABLE table=default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:34.989639 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d44e6e83e21f4c9485a4df3e40f8f72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:34.991694 4341 tablet_bootstrap.cc:492] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:34.993052 4337 tablet_bootstrap.cc:654] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:35.000892 4341 ts_tablet_manager.cc:1403] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.016s
I20260501 13:58:35.001003 4338 consensus_queue.cc:260] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.004107 4341 raft_consensus.cc:359] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.008872 4341 raft_consensus.cc:385] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:35.009621 4341 raft_consensus.cc:740] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:35.015467 4341 consensus_queue.cc:260] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.033629 4341 ts_tablet_manager.cc:1434] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.032s user 0.004s sys 0.001s
I20260501 13:58:35.037096 4338 ts_tablet_manager.cc:1434] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.102s user 0.005s sys 0.001s
I20260501 13:58:35.044616 4063 catalog_manager.cc:3502] renamed table default.table_ed51787806aa40069d50f787cc85ae8f [id=8d51c67732064782bd5d53da7178ba87] in HMS: new name default.table_e7ee55876f2f41cfa624a58f23f9d415
I20260501 13:58:35.057860 4338 tablet_bootstrap.cc:492] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:35.070031 4337 tablet_bootstrap.cc:492] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:35.070664 4337 ts_tablet_manager.cc:1403] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.191s user 0.009s sys 0.004s
I20260501 13:58:35.073469 4337 raft_consensus.cc:359] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.080699 4337 raft_consensus.cc:385] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:35.082214 4338 tablet_bootstrap.cc:654] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:35.083787 4341 tablet_bootstrap.cc:492] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:35.083696 4337 raft_consensus.cc:740] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:35.084666 4337 consensus_queue.cc:260] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.090317 4341 tablet_bootstrap.cc:654] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:35.094018 4309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.16.148:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45913
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.16.190:38561,127.1.16.189:44255,127.1.16.188:38253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.16.189:44255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.16.189
--webserver_port=36257
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type FASTDEBUG
built by None at 01 May 2026 13:43:23 UTC on e7f111948823
build id 11695
TSAN enabled
I20260501 13:58:35.096381 4309 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 13:58:35.098939 4309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260501 13:58:35.106166 4337 ts_tablet_manager.cc:1434] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.035s user 0.007s sys 0.001s
I20260501 13:58:35.107445 4337 tablet_bootstrap.cc:492] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:35.117902 4338 tablet_bootstrap.cc:492] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:35.118438 4338 ts_tablet_manager.cc:1403] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.001s
I20260501 13:58:35.121117 4338 raft_consensus.cc:359] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:35.126883 4341 tablet_bootstrap.cc:492] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:35.127424 4341 ts_tablet_manager.cc:1403] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.044s user 0.015s sys 0.005s
I20260501 13:58:35.127794 4338 raft_consensus.cc:385] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:35.128129 4338 raft_consensus.cc:740] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:35.128891 4338 consensus_queue.cc:260] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:35.131560 4341 raft_consensus.cc:359] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:35.134229 4338 ts_tablet_manager.cc:1434] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.015s user 0.001s sys 0.004s
I20260501 13:58:35.135094 4337 tablet_bootstrap.cc:654] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:35.135365 4341 raft_consensus.cc:385] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:35.136652 4341 raft_consensus.cc:740] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:35.137567 4341 consensus_queue.cc:260] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:35.154940 4341 ts_tablet_manager.cc:1434] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.027s user 0.003s sys 0.003s
I20260501 13:58:35.165302 4337 tablet_bootstrap.cc:492] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:35.165838 4337 ts_tablet_manager.cc:1403] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.005s
I20260501 13:58:35.168665 4337 raft_consensus.cc:359] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:35.171322 4350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:58:35.171805 4337 raft_consensus.cc:385] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:35.172219 4337 raft_consensus.cc:740] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:35.174801 4337 consensus_queue.cc:260] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:35.184384 4337 ts_tablet_manager.cc:1434] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260501 13:58:35.235900 4345 raft_consensus.cc:493] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:35.236423 4345 raft_consensus.cc:515] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.238263 4345 leader_election.cc:290] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:35.239950 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80a2f6fab87a4ad980c7816f34f41aff" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:35.240064 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80a2f6fab87a4ad980c7816f34f41aff" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:35.240562 3674 raft_consensus.cc:2468] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:35.240680 3821 raft_consensus.cc:2468] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:35.241786 3448 leader_election.cc:304] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:35.244731 4345 raft_consensus.cc:2804] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:35.245162 4345 raft_consensus.cc:493] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:35.245553 4345 raft_consensus.cc:3060] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:35.252203 4345 raft_consensus.cc:515] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.269711 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80a2f6fab87a4ad980c7816f34f41aff" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:35.270294 3674 raft_consensus.cc:3060] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:35.270375 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80a2f6fab87a4ad980c7816f34f41aff" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:35.270912 3821 raft_consensus.cc:3060] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:35.277241 3821 raft_consensus.cc:2468] T 80a2f6fab87a4ad980c7816f34f41aff P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:35.278836 3674 raft_consensus.cc:2468] T 80a2f6fab87a4ad980c7816f34f41aff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:35.280041 3448 leader_election.cc:304] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:35.285838 4345 leader_election.cc:290] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:35.286496 4345 raft_consensus.cc:2804] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:35.288455 4345 raft_consensus.cc:697] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:35.291934 4345 consensus_queue.cc:237] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.309370 4061 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]
I20260501 13:58:35.310009 4064 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129] alter complete (version 1)
W20260501 13:58:35.312856 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
W20260501 13:58:35.313462 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:58:35.313797 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet ba69864093504babbb00fb6170774778
W20260501 13:58:35.314009 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:35.314190 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:35.315604 4055 master_service.cc:954] ReplaceTablet: received request to replace tablet a9ecb5b51ae04f32a4f07f828d563fa4 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:35.316498 4055 master_service.cc:954] ReplaceTablet: received request to replace tablet d129cc8c1fc840cd9e00d4ad22ff236c from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:35.363973 4055 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet d129cc8c1fc840cd9e00d4ad22ff236c
I20260501 13:58:35.365160 4064 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:35.365986 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet d129cc8c1fc840cd9e00d4ad22ff236c with delete_type TABLET_DATA_DELETED (replaced by tablet 5b0433c7c6804a44965b54f04db94c70) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:35.368973 4247 raft_consensus.cc:493] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:35.369417 4247 raft_consensus.cc:515] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:35.371310 4247 leader_election.cc:290] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:35.372861 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d44e6e83e21f4c9485a4df3e40f8f72b" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:35.373433 3515 raft_consensus.cc:2468] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:35.375257 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d44e6e83e21f4c9485a4df3e40f8f72b" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:35.375871 3821 raft_consensus.cc:2468] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
W20260501 13:58:35.376086 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
I20260501 13:58:35.377178 3609 leader_election.cc:304] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:35.381318 4247 raft_consensus.cc:2804] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:35.381775 4247 raft_consensus.cc:493] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:35.381814 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet d129cc8c1fc840cd9e00d4ad22ff236c with delete_type TABLET_DATA_DELETED (replaced by tablet 5b0433c7c6804a44965b54f04db94c70) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:35.382383 4357 tablet_replica.cc:333] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:35.382912 4247 raft_consensus.cc:3060] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:35.383239 4357 raft_consensus.cc:2243] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:35.388898 4357 raft_consensus.cc:2272] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:35.390156 4055 catalog_manager.cc:6741] ReplaceTablet: tablet d129cc8c1fc840cd9e00d4ad22ff236c deleted and replaced by tablet 5b0433c7c6804a44965b54f04db94c70
I20260501 13:58:35.391404 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet d129cc8c1fc840cd9e00d4ad22ff236c with delete_type TABLET_DATA_DELETED (replaced by tablet 5b0433c7c6804a44965b54f04db94c70) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:35.392782 3651 tablet_service.cc:1558] Processing DeleteTablet for tablet d129cc8c1fc840cd9e00d4ad22ff236c with delete_type TABLET_DATA_DELETED (replaced by tablet 5b0433c7c6804a44965b54f04db94c70) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:35.401207 4357 ts_tablet_manager.cc:1916] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:35.408608 4247 raft_consensus.cc:515] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:35.421206 3493 tablet_service.cc:1511] Processing CreateTablet for tablet 5b0433c7c6804a44965b54f04db94c70 (DEFAULT_TABLE table=default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:35.422410 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0433c7c6804a44965b54f04db94c70. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:58:35.423734 4055 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
I20260501 13:58:35.417390 3651 tablet_service.cc:1511] Processing CreateTablet for tablet 5b0433c7c6804a44965b54f04db94c70 (DEFAULT_TABLE table=default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:35.431613 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0433c7c6804a44965b54f04db94c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:35.445771 4357 ts_tablet_manager.cc:1929] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260501 13:58:35.452632 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
I20260501 13:58:35.453747 4357 log.cc:1199] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/d129cc8c1fc840cd9e00d4ad22ff236c
I20260501 13:58:35.454692 4357 ts_tablet_manager.cc:1950] T d129cc8c1fc840cd9e00d4ad22ff236c P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:35.461580 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet d129cc8c1fc840cd9e00d4ad22ff236c (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:35.464072 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d44e6e83e21f4c9485a4df3e40f8f72b" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:35.464596 3515 raft_consensus.cc:3060] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:35.470340 3515 raft_consensus.cc:2468] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:35.471794 3609 leader_election.cc:304] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:35.475229 4361 raft_consensus.cc:2804] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:35.414758 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 5b0433c7c6804a44965b54f04db94c70 (DEFAULT_TABLE table=default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:35.487816 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0433c7c6804a44965b54f04db94c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:35.411557 4359 tablet_replica.cc:333] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:35.495349 4359 raft_consensus.cc:2243] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:35.495995 4359 raft_consensus.cc:2272] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:35.499886 4359 ts_tablet_manager.cc:1916] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:35.502267 4247 leader_election.cc:290] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:35.479844 4358 tablet_replica.cc:333] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:35.511090 4358 raft_consensus.cc:2243] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:35.512202 4358 raft_consensus.cc:2272] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:35.515077 4358 ts_tablet_manager.cc:1916] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:35.501811 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d44e6e83e21f4c9485a4df3e40f8f72b" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:35.520185 3821 raft_consensus.cc:3060] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:35.527714 3821 raft_consensus.cc:2468] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:35.547533 4338 tablet_bootstrap.cc:492] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:35.560091 4066 catalog_manager.cc:2174] created HMS catalog entry for table [id=748e365ce7f04451b1157e00a8813bb9]
I20260501 13:58:35.579952 4066 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 34) took 1148 ms. Trace:
I20260501 13:58:35.580482 4066 rpcz_store.cc:276] 0501 13:58:34.431797 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:34.450514 (+ 18717us) service_pool.cc:224] Handling call
0501 13:58:34.696013 (+245499us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:34.729353 (+ 33340us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:35.560042 (+830689us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:35.579032 (+ 18990us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:35.579418 (+ 386us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:35.579840 (+ 422us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:35.561992 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:35.562440 (+ 448us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:35.562490 (+ 50us) write_op.cc:435] Acquired schema lock
0501 13:58:35.562506 (+ 16us) tablet.cc:662] Decoding operations
0501 13:58:35.562807 (+ 301us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:35.562862 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:35.562885 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:35.563083 (+ 198us) tablet.cc:701] Row locks acquired
0501 13:58:35.563099 (+ 16us) write_op.cc:260] PREPARE: finished
0501 13:58:35.563232 (+ 133us) write_op.cc:270] Start()
0501 13:58:35.563330 (+ 98us) write_op.cc:276] Timestamp: P: 1777643915563188 usec, L: 0
0501 13:58:35.563343 (+ 13us) op_driver.cc:348] REPLICATION: starting
0501 13:58:35.564041 (+ 698us) log.cc:844] Serialized 432 byte log entry
0501 13:58:35.575993 (+ 11952us) op_driver.cc:464] REPLICATION: finished
0501 13:58:35.576581 (+ 588us) write_op.cc:301] APPLY: starting
0501 13:58:35.576643 (+ 62us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:35.576749 (+ 106us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:35.576832 (+ 83us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:35.577295 (+ 463us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:35.577426 (+ 131us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:35.577470 (+ 44us) write_op.cc:312] APPLY: finished
0501 13:58:35.577654 (+ 184us) log.cc:844] Serialized 28 byte log entry
0501 13:58:35.577812 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:35.577900 (+ 88us) write_op.cc:454] Released schema lock
0501 13:58:35.577939 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:35.578164 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:35.578188 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":579439,"Hive Metastore.run_cpu_time_us":684,"Hive Metastore.run_wall_time_us":250519,"child_traces":[["op",{"apply.queue_time_us":421,"apply.run_cpu_time_us":1549,"apply.run_wall_time_us":1549,"num_ops":2,"prepare.queue_time_us":135,"prepare.run_cpu_time_us":2313,"prepare.run_wall_time_us":2411,"raft.queue_time_us":3368,"raft.run_cpu_time_us":793,"raft.run_wall_time_us":792,"replication_time_us":12526}]]}
I20260501 13:58:35.587055 4337 tablet_bootstrap.cc:492] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:35.594337 4337 tablet_bootstrap.cc:654] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:35.601809 3608 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:38685, user_credentials={real_user=slave}} blocked reactor thread for 72830.7us
I20260501 13:58:35.603680 4341 tablet_bootstrap.cc:492] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:35.607223 4361 raft_consensus.cc:697] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:35.608192 4361 consensus_queue.cc:237] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:35.610924 4341 tablet_bootstrap.cc:654] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:35.610893 4337 tablet_bootstrap.cc:492] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:35.607225 4338 tablet_bootstrap.cc:654] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:35.611567 4337 ts_tablet_manager.cc:1403] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.002s
I20260501 13:58:35.615494 4337 raft_consensus.cc:359] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.618606 4337 raft_consensus.cc:385] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:35.623435 4337 raft_consensus.cc:740] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:35.628072 4341 tablet_bootstrap.cc:492] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:35.628490 4341 ts_tablet_manager.cc:1403] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.007s
I20260501 13:58:35.631172 4341 raft_consensus.cc:359] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.631830 4341 raft_consensus.cc:385] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:35.632094 4341 raft_consensus.cc:740] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:35.632790 4341 consensus_queue.cc:260] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.637322 4359 ts_tablet_manager.cc:1929] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:35.644057 4359 log.cc:1199] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/d129cc8c1fc840cd9e00d4ad22ff236c
I20260501 13:58:35.652143 4359 ts_tablet_manager.cc:1950] T d129cc8c1fc840cd9e00d4ad22ff236c P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:35.625054 4337 consensus_queue.cc:260] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.655802 4358 ts_tablet_manager.cc:1929] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:35.660989 4064 master_service.cc:954] ReplaceTablet: received request to replace tablet 7b9f3b4229594e13959fec0aa8281c03 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:35.661652 4358 log.cc:1199] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/d129cc8c1fc840cd9e00d4ad22ff236c
I20260501 13:58:35.662587 4358 ts_tablet_manager.cc:1950] T d129cc8c1fc840cd9e00d4ad22ff236c P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:35.659219 4337 ts_tablet_manager.cc:1434] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.047s user 0.005s sys 0.003s
I20260501 13:58:35.674307 4338 tablet_bootstrap.cc:492] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:35.677212 4338 ts_tablet_manager.cc:1403] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.130s user 0.011s sys 0.008s
I20260501 13:58:35.681324 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet d129cc8c1fc840cd9e00d4ad22ff236c (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
W20260501 13:58:35.682186 4036 catalog_manager.cc:4969] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet d129cc8c1fc840cd9e00d4ad22ff236c because the tablet was not found. No further retry: Not found: Tablet not found: d129cc8c1fc840cd9e00d4ad22ff236c
I20260501 13:58:35.682989 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet d129cc8c1fc840cd9e00d4ad22ff236c (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:35.685041 4054 catalog_manager.cc:2174] created HMS catalog entry for table [id=9535ed975c6d458f97c9175b0a34f3ab]
I20260501 13:58:35.685179 4338 raft_consensus.cc:359] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.685820 4338 raft_consensus.cc:385] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:35.686117 4338 raft_consensus.cc:740] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:35.695538 4338 consensus_queue.cc:260] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.698810 4055 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 5b0433c7c6804a44965b54f04db94c70 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]). Expected version 1 got 0
I20260501 13:58:35.703989 4054 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 39) took 1270 ms. Trace:
W20260501 13:58:35.703085 4055 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 5b0433c7c6804a44965b54f04db94c70 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 5b0433c7c6804a44965b54f04db94c70 consensus state has no leader
I20260501 13:58:35.691618 4341 ts_tablet_manager.cc:1434] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.063s user 0.005s sys 0.002s
I20260501 13:58:35.709671 4338 ts_tablet_manager.cc:1434] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260501 13:58:35.704337 4054 rpcz_store.cc:276] 0501 13:58:34.433467 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:34.441788 (+ 8321us) service_pool.cc:224] Handling call
0501 13:58:34.700029 (+258241us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:34.732378 (+ 32349us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:35.684994 (+952616us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:35.703234 (+ 18240us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:35.703587 (+ 353us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:35.703919 (+ 332us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:35.686954 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:35.687416 (+ 462us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:35.687445 (+ 29us) write_op.cc:435] Acquired schema lock
0501 13:58:35.687462 (+ 17us) tablet.cc:662] Decoding operations
0501 13:58:35.687788 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:35.687834 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:35.687858 (+ 24us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:35.688074 (+ 216us) tablet.cc:701] Row locks acquired
0501 13:58:35.688091 (+ 17us) write_op.cc:260] PREPARE: finished
0501 13:58:35.688222 (+ 131us) write_op.cc:270] Start()
0501 13:58:35.688311 (+ 89us) write_op.cc:276] Timestamp: P: 1777643915688177 usec, L: 0
0501 13:58:35.688331 (+ 20us) op_driver.cc:348] REPLICATION: starting
0501 13:58:35.688859 (+ 528us) log.cc:844] Serialized 432 byte log entry
0501 13:58:35.700936 (+ 12077us) op_driver.cc:464] REPLICATION: finished
0501 13:58:35.701373 (+ 437us) write_op.cc:301] APPLY: starting
0501 13:58:35.701442 (+ 69us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:35.701542 (+ 100us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:35.701668 (+ 126us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:35.702066 (+ 398us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:35.702209 (+ 143us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:35.702254 (+ 45us) write_op.cc:312] APPLY: finished
0501 13:58:35.702440 (+ 186us) log.cc:844] Serialized 28 byte log entry
0501 13:58:35.702618 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:35.702687 (+ 69us) write_op.cc:454] Released schema lock
0501 13:58:35.702719 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:35.702941 (+ 222us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:35.702965 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":827135,"Hive Metastore.run_cpu_time_us":576,"Hive Metastore.run_wall_time_us":124251,"child_traces":[["op",{"apply.queue_time_us":292,"apply.run_cpu_time_us":1532,"apply.run_wall_time_us":1532,"num_ops":2,"prepare.queue_time_us":120,"prepare.run_cpu_time_us":2244,"prepare.run_wall_time_us":2247,"raft.queue_time_us":808,"raft.run_cpu_time_us":716,"raft.run_wall_time_us":715,"replication_time_us":12505}]]}
I20260501 13:58:35.730408 4055 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 5b0433c7c6804a44965b54f04db94c70 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]). Expected version 1 got 0
I20260501 13:58:35.757404 4064 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet 7b9f3b4229594e13959fec0aa8281c03
I20260501 13:58:35.764690 3651 tablet_service.cc:1558] Processing DeleteTablet for tablet 7b9f3b4229594e13959fec0aa8281c03 with delete_type TABLET_DATA_DELETED (replaced by tablet 02102952baaf40e0b897bc6dc04426da) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:35.766439 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 7b9f3b4229594e13959fec0aa8281c03 with delete_type TABLET_DATA_DELETED (replaced by tablet 02102952baaf40e0b897bc6dc04426da) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:35.766763 4064 catalog_manager.cc:6741] ReplaceTablet: tablet 7b9f3b4229594e13959fec0aa8281c03 deleted and replaced by tablet 02102952baaf40e0b897bc6dc04426da
I20260501 13:58:35.768321 4064 master_service.cc:954] ReplaceTablet: received request to replace tablet 80a2f6fab87a4ad980c7816f34f41aff from {username='slave'} at 127.0.0.1:35438
W20260501 13:58:35.769124 4036 catalog_manager.cc:4969] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet 7b9f3b4229594e13959fec0aa8281c03 because the tablet was not found. No further retry: Not found: Tablet not found: 7b9f3b4229594e13959fec0aa8281c03
W20260501 13:58:35.769838 4036 catalog_manager.cc:4969] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet 7b9f3b4229594e13959fec0aa8281c03 because the tablet was not found. No further retry: Not found: Tablet not found: 7b9f3b4229594e13959fec0aa8281c03
W20260501 13:58:35.772940 4037 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 5b0433c7c6804a44965b54f04db94c70 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5b0433c7c6804a44965b54f04db94c70 consensus state has no leader
I20260501 13:58:35.775048 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet 7b9f3b4229594e13959fec0aa8281c03 with delete_type TABLET_DATA_DELETED (replaced by tablet 02102952baaf40e0b897bc6dc04426da) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:35.776405 4357 tablet_replica.cc:333] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:35.777320 4357 raft_consensus.cc:2243] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:35.777997 4357 raft_consensus.cc:2272] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:35.780638 4357 ts_tablet_manager.cc:1916] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:35.787791 4054 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 5b0433c7c6804a44965b54f04db94c70 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]). Expected version 1 got 0
I20260501 13:58:35.791631 4066 catalog_manager.cc:5671] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:35.799758 3651 tablet_service.cc:1511] Processing CreateTablet for tablet cf6482b9e6994782b0b1ca4250af8e97 (DEFAULT_TABLE table=default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:35.800895 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6482b9e6994782b0b1ca4250af8e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:35.809288 4345 raft_consensus.cc:493] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:35.809906 4345 raft_consensus.cc:515] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.812148 4345 leader_election.cc:290] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:35.813908 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b0433c7c6804a44965b54f04db94c70" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:35.814565 3821 raft_consensus.cc:2468] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:35.815498 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b0433c7c6804a44965b54f04db94c70" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:35.815855 3448 leader_election.cc:304] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:35.816118 3674 raft_consensus.cc:2468] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:35.818885 4345 raft_consensus.cc:2804] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:35.819258 4345 raft_consensus.cc:493] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:35.819634 4345 raft_consensus.cc:3060] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:35.834697 3493 tablet_service.cc:1511] Processing CreateTablet for tablet cf6482b9e6994782b0b1ca4250af8e97 (DEFAULT_TABLE table=default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:35.836357 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6482b9e6994782b0b1ca4250af8e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:35.852981 3802 tablet_service.cc:1511] Processing CreateTablet for tablet cf6482b9e6994782b0b1ca4250af8e97 (DEFAULT_TABLE table=default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:35.853987 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6482b9e6994782b0b1ca4250af8e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:35.795840 4064 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_b4b1da6bbecc440286963f934507777c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:35.862397 4345 raft_consensus.cc:515] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.864995 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b0433c7c6804a44965b54f04db94c70" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:35.865562 3821 raft_consensus.cc:3060] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:35.869232 4338 tablet_bootstrap.cc:492] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:35.876153 4338 tablet_bootstrap.cc:654] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:35.877925 3821 raft_consensus.cc:2468] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:35.879882 3448 leader_election.cc:304] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:35.895874 4370 raft_consensus.cc:2804] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:35.897908 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b0433c7c6804a44965b54f04db94c70" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:35.898499 3674 raft_consensus.cc:3060] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:35.905846 3674 raft_consensus.cc:2468] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:35.907895 4345 leader_election.cc:290] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:35.915426 4338 tablet_bootstrap.cc:492] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:35.915951 4338 ts_tablet_manager.cc:1403] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.000s
I20260501 13:58:35.918973 4066 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_0da08f161093429e83b826825c3a9727"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:35.921030 4341 tablet_bootstrap.cc:492] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:35.922547 4059 catalog_manager.cc:2174] created HMS catalog entry for table [id=53fa808a27054cdf834732393dc08838]
I20260501 13:58:35.924851 4337 tablet_bootstrap.cc:492] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:35.927299 4341 tablet_bootstrap.cc:654] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:35.934194 4337 tablet_bootstrap.cc:654] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:35.938632 4338 raft_consensus.cc:359] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.939348 4338 raft_consensus.cc:385] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:35.939639 4338 raft_consensus.cc:740] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:35.943007 4055 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 5b0433c7c6804a44965b54f04db94c70 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]). Expected version 1 got 0
I20260501 13:58:35.946152 4054 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 5b0433c7c6804a44965b54f04db94c70 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]). Expected version 1 got 0
I20260501 13:58:35.954500 4341 tablet_bootstrap.cc:492] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:35.954962 4341 ts_tablet_manager.cc:1403] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.006s
I20260501 13:58:35.955843 3651 tablet_service.cc:1511] Processing CreateTablet for tablet 02102952baaf40e0b897bc6dc04426da (DEFAULT_TABLE table=default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:35.956950 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02102952baaf40e0b897bc6dc04426da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:35.959810 4341 raft_consensus.cc:359] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.960520 4341 raft_consensus.cc:385] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:35.960837 4341 raft_consensus.cc:740] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:35.961000 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 02102952baaf40e0b897bc6dc04426da (DEFAULT_TABLE table=default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:35.964854 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02102952baaf40e0b897bc6dc04426da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:35.967914 4059 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 36) took 1535 ms. Trace:
I20260501 13:58:35.968302 4059 rpcz_store.cc:276] 0501 13:58:34.432505 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:34.448236 (+ 15731us) service_pool.cc:224] Handling call
0501 13:58:34.704275 (+256039us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:34.734842 (+ 30567us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:35.922506 (+1187664us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:35.967217 (+ 44711us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:35.967549 (+ 332us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:35.967849 (+ 300us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:35.930098 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:35.930595 (+ 497us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:35.930623 (+ 28us) write_op.cc:435] Acquired schema lock
0501 13:58:35.930635 (+ 12us) tablet.cc:662] Decoding operations
0501 13:58:35.930945 (+ 310us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:35.930985 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:35.931009 (+ 24us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:35.931220 (+ 211us) tablet.cc:701] Row locks acquired
0501 13:58:35.931233 (+ 13us) write_op.cc:260] PREPARE: finished
0501 13:58:35.931364 (+ 131us) write_op.cc:270] Start()
0501 13:58:35.931459 (+ 95us) write_op.cc:276] Timestamp: P: 1777643915931307 usec, L: 0
0501 13:58:35.931478 (+ 19us) op_driver.cc:348] REPLICATION: starting
0501 13:58:35.932190 (+ 712us) log.cc:844] Serialized 433 byte log entry
0501 13:58:35.964943 (+ 32753us) op_driver.cc:464] REPLICATION: finished
0501 13:58:35.965493 (+ 550us) write_op.cc:301] APPLY: starting
0501 13:58:35.965564 (+ 71us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:35.965692 (+ 128us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:35.965794 (+ 102us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:35.966240 (+ 446us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:35.966403 (+ 163us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:35.966454 (+ 51us) write_op.cc:312] APPLY: finished
0501 13:58:35.966714 (+ 260us) log.cc:844] Serialized 29 byte log entry
0501 13:58:35.966894 (+ 180us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:35.966974 (+ 80us) write_op.cc:454] Released schema lock
0501 13:58:35.967011 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:35.967246 (+ 235us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:35.967268 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":945893,"Hive Metastore.run_cpu_time_us":592,"Hive Metastore.run_wall_time_us":237712,"child_traces":[["op",{"apply.queue_time_us":383,"apply.run_cpu_time_us":1725,"apply.run_wall_time_us":1724,"mutex_wait_us":5548,"num_ops":2,"prepare.queue_time_us":5692,"prepare.run_cpu_time_us":2254,"prepare.run_wall_time_us":2565,"raft.queue_time_us":82,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":15,"replication_time_us":33368}]]}
I20260501 13:58:35.961812 4338 consensus_queue.cc:260] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.981051 3493 tablet_service.cc:1511] Processing CreateTablet for tablet 02102952baaf40e0b897bc6dc04426da (DEFAULT_TABLE table=default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:35.982170 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02102952baaf40e0b897bc6dc04426da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:35.982540 4370 raft_consensus.cc:697] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:35.974917 4341 consensus_queue.cc:260] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:35.989197 4338 ts_tablet_manager.cc:1434] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.073s user 0.007s sys 0.000s
I20260501 13:58:35.998800 4341 ts_tablet_manager.cc:1434] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260501 13:58:36.003006 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 1505e4bc749b48a88008d732881ff148 (DEFAULT_TABLE table=default.table_06e2e16b792549e68d2e01ce3e3ce8b5 [id=9535ed975c6d458f97c9175b0a34f3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.003711 3801 tablet_service.cc:1511] Processing CreateTablet for tablet 1505e4bc749b48a88008d732881ff148 (DEFAULT_TABLE table=default.table_06e2e16b792549e68d2e01ce3e3ce8b5 [id=9535ed975c6d458f97c9175b0a34f3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.004062 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1505e4bc749b48a88008d732881ff148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:36.004670 3801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1505e4bc749b48a88008d732881ff148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:35.983528 4370 consensus_queue.cc:237] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.020433 3491 tablet_service.cc:1511] Processing CreateTablet for tablet 1505e4bc749b48a88008d732881ff148 (DEFAULT_TABLE table=default.table_06e2e16b792549e68d2e01ce3e3ce8b5 [id=9535ed975c6d458f97c9175b0a34f3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.021595 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1505e4bc749b48a88008d732881ff148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:36.029273 4341 tablet_bootstrap.cc:492] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:35.964546 4357 ts_tablet_manager.cc:1929] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:36.038985 4357 log.cc:1199] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/7b9f3b4229594e13959fec0aa8281c03
I20260501 13:58:36.040038 4357 ts_tablet_manager.cc:1950] T 7b9f3b4229594e13959fec0aa8281c03 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:36.041568 4187 catalog_manager.cc:2772] Deleting table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129] in response to Hive Metastore notification log event 15
I20260501 13:58:36.041852 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:36.052280 4337 tablet_bootstrap.cc:492] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:36.052832 4337 ts_tablet_manager.cc:1403] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.128s user 0.009s sys 0.012s
I20260501 13:58:36.056125 4341 tablet_bootstrap.cc:654] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.067104 4337 raft_consensus.cc:359] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.073690 4341 tablet_bootstrap.cc:492] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:36.074146 4341 ts_tablet_manager.cc:1403] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.000s
I20260501 13:58:36.078303 4337 raft_consensus.cc:385] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.079115 4337 raft_consensus.cc:740] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.079855 4337 consensus_queue.cc:260] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.082005 4337 ts_tablet_manager.cc:1434] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.029s user 0.004s sys 0.000s
I20260501 13:58:36.092078 4341 raft_consensus.cc:359] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.092783 4341 raft_consensus.cc:385] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.093055 4341 raft_consensus.cc:740] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.093662 4341 consensus_queue.cc:260] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.101671 4187 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet 5b0433c7c6804a44965b54f04db94c70
I20260501 13:58:36.110327 3492 tablet_service.cc:1558] Processing DeleteTablet for tablet 5b0433c7c6804a44965b54f04db94c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:36 UTC) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:36.112253 4332 raft_consensus.cc:493] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:36.112852 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 5b0433c7c6804a44965b54f04db94c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:36 UTC) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:36.112852 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 5b0433c7c6804a44965b54f04db94c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:36 UTC) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:36.112689 4332 raft_consensus.cc:515] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.113513 4359 tablet_replica.cc:333] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:36.113511 4358 tablet_replica.cc:333] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:36.114306 4359 raft_consensus.cc:2243] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:36.114290 4358 raft_consensus.cc:2243] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:36.114775 4358 raft_consensus.cc:2272] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:36.114784 4359 raft_consensus.cc:2272] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:36.121757 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6482b9e6994782b0b1ca4250af8e97" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:36.122376 3672 raft_consensus.cc:2468] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:36.123626 3756 leader_election.cc:304] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:36.124984 4358 ts_tablet_manager.cc:1916] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:36.131153 4359 ts_tablet_manager.cc:1916] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:36.134682 4332 leader_election.cc:290] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:36.135640 4332 raft_consensus.cc:2804] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:36.136430 4332 raft_consensus.cc:493] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:36.137087 4332 raft_consensus.cc:3060] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:36.144119 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6482b9e6994782b0b1ca4250af8e97" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:36.144822 3515 raft_consensus.cc:2468] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:36.151683 4332 raft_consensus.cc:515] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.159687 4332 leader_election.cc:290] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:36.161007 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6482b9e6994782b0b1ca4250af8e97" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:36.161631 3513 raft_consensus.cc:3060] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:36.165604 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf6482b9e6994782b0b1ca4250af8e97" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:36.166160 3672 raft_consensus.cc:3060] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:36.170728 4052 catalog_manager.cc:2174] created HMS catalog entry for table [id=f6a8e3a12e48471fb8788a66cd890ea2]
I20260501 13:58:36.156519 4341 ts_tablet_manager.cc:1434] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.082s user 0.007s sys 0.000s
I20260501 13:58:36.176512 4341 tablet_bootstrap.cc:492] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:36.179646 4055 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_c0462a887c864245890f76dea029e11b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:36.197256 3654 tablet_service.cc:1511] Processing CreateTablet for tablet fd4467544b3743a8acb44473f0b35203 (DEFAULT_TABLE table=default.table_f52f85ad11fb4023adead2ab9c5eb694 [id=53fa808a27054cdf834732393dc08838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.197327 3492 tablet_service.cc:1511] Processing CreateTablet for tablet fd4467544b3743a8acb44473f0b35203 (DEFAULT_TABLE table=default.table_f52f85ad11fb4023adead2ab9c5eb694 [id=53fa808a27054cdf834732393dc08838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.198026 4358 ts_tablet_manager.cc:1929] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:36.198390 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd4467544b3743a8acb44473f0b35203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:36.198448 3492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd4467544b3743a8acb44473f0b35203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:36.199107 3802 tablet_service.cc:1511] Processing CreateTablet for tablet fd4467544b3743a8acb44473f0b35203 (DEFAULT_TABLE table=default.table_f52f85ad11fb4023adead2ab9c5eb694 [id=53fa808a27054cdf834732393dc08838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.200132 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd4467544b3743a8acb44473f0b35203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:36.210799 4054 master_service.cc:954] ReplaceTablet: received request to replace tablet 80a2f6fab87a4ad980c7816f34f41aff from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:36.211954 4059 master_service.cc:954] ReplaceTablet: received request to replace tablet 80a2f6fab87a4ad980c7816f34f41aff from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:36.215997 4359 ts_tablet_manager.cc:1929] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:36.219066 4052 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 40) took 1785 ms. Trace:
I20260501 13:58:36.219427 4052 rpcz_store.cc:276] 0501 13:58:34.433763 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:34.440566 (+ 6803us) service_pool.cc:224] Handling call
0501 13:58:34.692194 (+251628us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:34.743979 (+ 51785us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:36.170680 (+1426701us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:36.218280 (+ 47600us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:36.218671 (+ 391us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:36.218991 (+ 320us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:36.172601 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:36.173064 (+ 463us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:36.173091 (+ 27us) write_op.cc:435] Acquired schema lock
0501 13:58:36.173104 (+ 13us) tablet.cc:662] Decoding operations
0501 13:58:36.173420 (+ 316us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:36.173459 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:36.173485 (+ 26us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:36.173701 (+ 216us) tablet.cc:701] Row locks acquired
0501 13:58:36.173713 (+ 12us) write_op.cc:260] PREPARE: finished
0501 13:58:36.173841 (+ 128us) write_op.cc:270] Start()
0501 13:58:36.173934 (+ 93us) write_op.cc:276] Timestamp: P: 1777643916173791 usec, L: 0
0501 13:58:36.173954 (+ 20us) op_driver.cc:348] REPLICATION: starting
0501 13:58:36.174539 (+ 585us) log.cc:844] Serialized 433 byte log entry
0501 13:58:36.216293 (+ 41754us) op_driver.cc:464] REPLICATION: finished
0501 13:58:36.216766 (+ 473us) write_op.cc:301] APPLY: starting
0501 13:58:36.216839 (+ 73us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:36.216948 (+ 109us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:36.217036 (+ 88us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:36.217419 (+ 383us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:36.217562 (+ 143us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:36.217603 (+ 41us) write_op.cc:312] APPLY: finished
0501 13:58:36.217805 (+ 202us) log.cc:844] Serialized 29 byte log entry
0501 13:58:36.217985 (+ 180us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:36.218056 (+ 71us) write_op.cc:454] Released schema lock
0501 13:58:36.218088 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":1217152,"Hive Metastore.run_cpu_time_us":599,"Hive Metastore.run_wall_time_us":208575,"spinlock_wait_cycles":45568,"child_traces":[["op",{"apply.queue_time_us":323,"num_ops":2,"prepare.queue_time_us":127,"prepare.run_cpu_time_us":2284,"prepare.run_wall_time_us":2293,"raft.queue_time_us":91,"raft.run_cpu_time_us":21,"raft.run_wall_time_us":17,"replication_time_us":42232,"spinlock_wait_cycles":31616}]]}
I20260501 13:58:36.224862 4386 tablet.cc:1724] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:36.233659 3672 raft_consensus.cc:2468] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:36.198390 4358 log.cc:1199] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/5b0433c7c6804a44965b54f04db94c70
I20260501 13:58:36.236372 3513 raft_consensus.cc:2468] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:36.237021 4358 ts_tablet_manager.cc:1950] T 5b0433c7c6804a44965b54f04db94c70 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:36.236091 4359 log.cc:1199] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/5b0433c7c6804a44965b54f04db94c70
I20260501 13:58:36.240620 4359 ts_tablet_manager.cc:1950] T 5b0433c7c6804a44965b54f04db94c70 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:36.240739 4338 tablet_bootstrap.cc:492] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:36.231465 4387 tablet.cc:1724] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:36.249404 3756 leader_election.cc:304] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:36.231854 4341 tablet_bootstrap.cc:654] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.250166 4332 raft_consensus.cc:2804] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:36.251222 4332 raft_consensus.cc:697] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:36.251989 4332 consensus_queue.cc:237] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.254889 4357 tablet_replica.cc:333] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:36.255815 4357 raft_consensus.cc:2243] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:36.256560 4357 pending_rounds.cc:70] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: Trying to abort 1 pending ops.
I20260501 13:58:36.257022 4357 pending_rounds.cc:77] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229480030138368 op_type: NO_OP noop_request { }
I20260501 13:58:36.257445 4357 raft_consensus.cc:2889] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:36.257789 4357 raft_consensus.cc:2272] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:58:36.260596 4052 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:36.262580 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 5b0433c7c6804a44965b54f04db94c70 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:36.263069 4337 tablet_bootstrap.cc:492] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:36.271207 3492 tablet_service.cc:1511] Processing CreateTablet for tablet 528c2d886bab4375b79f969343d9f46b (DEFAULT_TABLE table=default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.272358 3492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 528c2d886bab4375b79f969343d9f46b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:36.291407 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 5b0433c7c6804a44965b54f04db94c70 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:36.293910 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 528c2d886bab4375b79f969343d9f46b (DEFAULT_TABLE table=default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.293911 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 528c2d886bab4375b79f969343d9f46b (DEFAULT_TABLE table=default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.294955 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 528c2d886bab4375b79f969343d9f46b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:36.295912 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 528c2d886bab4375b79f969343d9f46b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:36.306201 4341 tablet_bootstrap.cc:492] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:36.306695 4341 ts_tablet_manager.cc:1403] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.130s user 0.008s sys 0.005s
I20260501 13:58:36.316463 4052 catalog_manager.cc:5671] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:36.319509 4357 ts_tablet_manager.cc:1916] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:36.328771 4370 consensus_queue.cc:1048] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:36.356125 4337 tablet_bootstrap.cc:654] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.357720 4341 raft_consensus.cc:359] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.308693 4338 tablet_bootstrap.cc:654] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.366365 4341 raft_consensus.cc:385] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.366959 4341 raft_consensus.cc:740] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.370347 4341 consensus_queue.cc:260] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.374032 4380 consensus_queue.cc:1048] T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260501 13:58:36.381320 4054 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
I20260501 13:58:36.386636 4337 tablet_bootstrap.cc:492] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:36.387151 4337 ts_tablet_manager.cc:1403] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.124s user 0.010s sys 0.006s
I20260501 13:58:36.391075 4337 raft_consensus.cc:359] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.391811 4337 raft_consensus.cc:385] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.392138 4337 raft_consensus.cc:740] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.392964 4337 consensus_queue.cc:260] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.395459 4337 ts_tablet_manager.cc:1434] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260501 13:58:36.396430 4337 tablet_bootstrap.cc:492] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:36.406917 4337 tablet_bootstrap.cc:654] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.433584 4059 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87] alter complete (version 1)
I20260501 13:58:36.434919 4059 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_124cdf87baa148fbbc6e5f06d1efcda1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:36.437142 4341 ts_tablet_manager.cc:1434] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.110s user 0.007s sys 0.000s
I20260501 13:58:36.444854 4397 tablet.cc:1724] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:36.444937 4341 tablet_bootstrap.cc:492] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:36.467909 4056 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]
I20260501 13:58:36.442797 4054 master_service.cc:954] ReplaceTablet: received request to replace tablet cf6482b9e6994782b0b1ca4250af8e97 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:36.473465 4341 tablet_bootstrap.cc:654] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.473519 4337 tablet_bootstrap.cc:492] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:36.474175 4337 ts_tablet_manager.cc:1403] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.078s user 0.012s sys 0.000s
I20260501 13:58:36.477753 4337 raft_consensus.cc:359] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
W20260501 13:58:36.480935 4392 log.cc:927] Time spent T 80a2f6fab87a4ad980c7816f34f41aff P 9154033cea374a10a5225c53b8bcf122: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20260501 13:58:36.482496 4379 raft_consensus.cc:493] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:36.483044 4379 raft_consensus.cc:515] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.486639 4379 leader_election.cc:290] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:36.489876 4382 consensus_queue.cc:1048] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:36.490099 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1505e4bc749b48a88008d732881ff148" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:36.479835 4337 raft_consensus.cc:385] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260501 13:58:36.491433 3757 leader_election.cc:343] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:36.491535 4337 raft_consensus.cc:740] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.492221 4337 consensus_queue.cc:260] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.494908 4337 ts_tablet_manager.cc:1434] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260501 13:58:36.506282 4187 catalog_manager.cc:2772] Deleting table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee] in response to Hive Metastore notification log event 20
I20260501 13:58:36.487365 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1505e4bc749b48a88008d732881ff148" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
W20260501 13:58:36.511535 3756 leader_election.cc:343] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:36.514403 3756 leader_election.cc:304] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122
W20260501 13:58:36.523936 4052 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
I20260501 13:58:36.541992 4332 raft_consensus.cc:2749] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:36.550918 4357 ts_tablet_manager.cc:1929] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:36.552904 4357 log.cc:1199] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/5b0433c7c6804a44965b54f04db94c70
I20260501 13:58:36.556070 4357 ts_tablet_manager.cc:1950] T 5b0433c7c6804a44965b54f04db94c70 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:36.556070 4187 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet d44e6e83e21f4c9485a4df3e40f8f72b
I20260501 13:58:36.557322 4382 consensus_queue.cc:1048] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:58:36.558912 4064 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b4b1da6bbecc440286963f934507777c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:36.559530 4061 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35438 (request call id 32) took 2186 ms. Trace:
I20260501 13:58:36.560978 4063 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35438 (request call id 33) took 2129 ms. Trace:
I20260501 13:58:36.559849 4061 rpcz_store.cc:276] 0501 13:58:34.373516 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:34.373660 (+ 144us) service_pool.cc:224] Handling call
0501 13:58:34.704725 (+331065us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:36.559460 (+1854735us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":338964,"Hive Metastore.run_cpu_time_us":394,"Hive Metastore.run_wall_time_us":264965}
W20260501 13:58:36.562738 4066 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0da08f161093429e83b826825c3a9727 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:36.564002 4061 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
I20260501 13:58:36.568238 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet d44e6e83e21f4c9485a4df3e40f8f72b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:36 UTC) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:36.568921 4359 tablet_replica.cc:333] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:36.569178 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet d44e6e83e21f4c9485a4df3e40f8f72b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:36 UTC) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:36.569710 4359 raft_consensus.cc:2243] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:36.561308 4063 rpcz_store.cc:276] 0501 13:58:34.431332 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:34.449292 (+ 17960us) service_pool.cc:224] Handling call
0501 13:58:34.701697 (+252405us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:36.557971 (+1856274us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:36.560015 (+ 2044us) catalog_manager.cc:3725] Apply alter schema
0501 13:58:36.560902 (+ 887us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":7457,"Hive Metastore.run_cpu_time_us":2155,"Hive Metastore.run_wall_time_us":334018}
I20260501 13:58:36.570204 4359 raft_consensus.cc:2272] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:36.571086 4358 tablet_replica.cc:333] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
W20260501 13:58:36.571786 4063 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
I20260501 13:58:36.571915 4358 raft_consensus.cc:2243] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:36.572784 4358 raft_consensus.cc:2272] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:36.572889 4063 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b0433c7c6804a44965b54f04db94c70 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:36.505789 4341 tablet_bootstrap.cc:492] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:36.576089 4341 ts_tablet_manager.cc:1403] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.131s user 0.012s sys 0.000s
I20260501 13:58:36.576804 4358 ts_tablet_manager.cc:1916] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:36.578574 4061 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
W20260501 13:58:36.582253 4037 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 -> Peer 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255): Couldn't send request to peer 15dd70d7c5494c3c911f0a780dc9a2af. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260501 13:58:36.585940 4359 ts_tablet_manager.cc:1916] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:36.588657 4338 tablet_bootstrap.cc:492] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:36.589162 4338 ts_tablet_manager.cc:1403] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.349s user 0.015s sys 0.004s
I20260501 13:58:36.589006 4054 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet cf6482b9e6994782b0b1ca4250af8e97
W20260501 13:58:36.589775 4061 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 02102952baaf40e0b897bc6dc04426da consensus state has no leader
I20260501 13:58:36.591162 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet cf6482b9e6994782b0b1ca4250af8e97 with delete_type TABLET_DATA_DELETED (replaced by tablet d5862fb4c6ca43feb61939b8698e728c) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:36.593097 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet cf6482b9e6994782b0b1ca4250af8e97 with delete_type TABLET_DATA_DELETED (replaced by tablet d5862fb4c6ca43feb61939b8698e728c) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:36.557881 3491 tablet_service.cc:1558] Processing DeleteTablet for tablet d44e6e83e21f4c9485a4df3e40f8f72b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:36 UTC) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:36.572139 4337 tablet_bootstrap.cc:492] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:36.593878 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet cf6482b9e6994782b0b1ca4250af8e97 with delete_type TABLET_DATA_DELETED (replaced by tablet d5862fb4c6ca43feb61939b8698e728c) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:36.597236 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 5b0433c7c6804a44965b54f04db94c70 (table default.table_2840ee60b88a494f9b40b448d78b0000 [id=0f9b011bb04e4620a47cd4853e6c4129]) successfully deleted
I20260501 13:58:36.599704 4054 catalog_manager.cc:6741] ReplaceTablet: tablet cf6482b9e6994782b0b1ca4250af8e97 deleted and replaced by tablet d5862fb4c6ca43feb61939b8698e728c
I20260501 13:58:36.599452 4338 raft_consensus.cc:359] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.600268 4338 raft_consensus.cc:385] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.600557 4338 raft_consensus.cc:740] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.602706 4341 raft_consensus.cc:359] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.603452 4341 raft_consensus.cc:385] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.603755 4341 raft_consensus.cc:740] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.604441 4341 consensus_queue.cc:260] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.605399 4357 tablet_replica.cc:333] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:36.606722 4338 consensus_queue.cc:260] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.613876 4337 tablet_bootstrap.cc:654] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.614284 4357 raft_consensus.cc:2243] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:36.614892 4357 raft_consensus.cc:2272] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:36.622910 4380 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:36.623404 4380 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:36.625345 4380 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:36.631152 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 13:58:36.631752 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:58:36.632090 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:36.634989 4359 ts_tablet_manager.cc:1929] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:36.635370 4359 log.cc:1199] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/d44e6e83e21f4c9485a4df3e40f8f72b
I20260501 13:58:36.636118 4359 ts_tablet_manager.cc:1950] T d44e6e83e21f4c9485a4df3e40f8f72b P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:36.640005 4380 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:36.640869 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet d44e6e83e21f4c9485a4df3e40f8f72b (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:36.648715 4359 tablet_replica.cc:333] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:36.654006 4341 ts_tablet_manager.cc:1434] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.078s user 0.006s sys 0.001s
I20260501 13:58:36.655138 4338 ts_tablet_manager.cc:1434] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.066s user 0.009s sys 0.000s
I20260501 13:58:36.659087 4359 raft_consensus.cc:2243] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:36.653785 4337 tablet_bootstrap.cc:492] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:36.663134 4337 ts_tablet_manager.cc:1403] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.091s user 0.013s sys 0.000s
I20260501 13:58:36.668308 3654 tablet_service.cc:1511] Processing CreateTablet for tablet d5862fb4c6ca43feb61939b8698e728c (DEFAULT_TABLE table=default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.669365 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5862fb4c6ca43feb61939b8698e728c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:36.669685 3493 tablet_service.cc:1511] Processing CreateTablet for tablet d5862fb4c6ca43feb61939b8698e728c (DEFAULT_TABLE table=default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.670763 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5862fb4c6ca43feb61939b8698e728c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:36.666095 4337 raft_consensus.cc:359] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.674980 4337 raft_consensus.cc:385] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.675288 4337 raft_consensus.cc:740] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.676059 4337 consensus_queue.cc:260] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.686784 4338 tablet_bootstrap.cc:492] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:36.688722 4337 ts_tablet_manager.cc:1434] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260501 13:58:36.690023 4063 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:36.692754 4359 pending_rounds.cc:70] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: Trying to abort 1 pending ops.
I20260501 13:58:36.693053 4359 pending_rounds.cc:77] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229480977670144 op_type: NO_OP noop_request { }
I20260501 13:58:36.693408 4359 raft_consensus.cc:2889] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:36.693778 4359 raft_consensus.cc:2272] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:36.693701 4382 raft_consensus.cc:493] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:36.694165 4382 raft_consensus.cc:515] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.696079 4382 leader_election.cc:290] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:36.699898 4341 tablet_bootstrap.cc:492] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:36.695715 4337 tablet_bootstrap.cc:492] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:36.702172 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1505e4bc749b48a88008d732881ff148" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:36.702796 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1505e4bc749b48a88008d732881ff148" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:36.702858 3822 raft_consensus.cc:2468] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:36.704088 3608 leader_election.cc:304] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
W20260501 13:58:36.704670 3609 leader_election.cc:343] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:36.706645 4382 raft_consensus.cc:2804] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:36.707036 4382 raft_consensus.cc:493] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:36.707441 4382 raft_consensus.cc:3060] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:36.712482 4359 ts_tablet_manager.cc:1916] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:36.716538 4380 raft_consensus.cc:493] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:36.717023 4380 raft_consensus.cc:515] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.717705 4382 raft_consensus.cc:515] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.718248 4338 tablet_bootstrap.cc:654] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.718812 4061 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:36.718940 4380 leader_election.cc:290] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:36.721812 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1505e4bc749b48a88008d732881ff148" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:36.722380 3822 raft_consensus.cc:3060] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
W20260501 13:58:36.647476 4402 consensus_queue.cc:1175] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9154033cea374a10a5225c53b8bcf122" status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260501 13:58:36.725502 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02102952baaf40e0b897bc6dc04426da" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:36.726100 3821 raft_consensus.cc:2468] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:36.727500 3448 leader_election.cc:304] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:36.727780 4357 ts_tablet_manager.cc:1916] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:36.728355 4380 raft_consensus.cc:2804] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:36.728778 4380 raft_consensus.cc:493] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:36.729123 4380 raft_consensus.cc:3060] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:36.738651 4341 tablet_bootstrap.cc:654] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.740955 4337 tablet_bootstrap.cc:654] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.743762 3822 raft_consensus.cc:2468] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:36.720798 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02102952baaf40e0b897bc6dc04426da" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:36.745179 3608 leader_election.cc:304] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:36.747437 4361 raft_consensus.cc:2804] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
W20260501 13:58:36.748486 3448 leader_election.cc:343] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:36.748436 4380 raft_consensus.cc:515] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.751636 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02102952baaf40e0b897bc6dc04426da" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
W20260501 13:58:36.753041 3448 leader_election.cc:343] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:36.754796 4345 raft_consensus.cc:493] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:36.755246 4345 raft_consensus.cc:515] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.756902 4341 tablet_bootstrap.cc:492] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:36.757061 4345 leader_election.cc:290] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:36.757341 4341 ts_tablet_manager.cc:1403] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.004s
I20260501 13:58:36.758268 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "528c2d886bab4375b79f969343d9f46b" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:36.763525 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "528c2d886bab4375b79f969343d9f46b" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
W20260501 13:58:36.760583 3448 leader_election.cc:343] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:36.767519 4361 raft_consensus.cc:697] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:36.768363 4361 consensus_queue.cc:237] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
W20260501 13:58:36.769191 3448 leader_election.cc:343] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:36.769621 3448 leader_election.cc:304] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:36.770341 4345 raft_consensus.cc:2749] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:36.770944 4063 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_e7ee55876f2f41cfa624a58f23f9d415" } modify_external_catalogs: true
I20260501 13:58:36.771631 4063 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_e7ee55876f2f41cfa624a58f23f9d415" } modify_external_catalogs: true
I20260501 13:58:36.774639 4341 raft_consensus.cc:359] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.775271 4341 raft_consensus.cc:385] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.776877 4357 ts_tablet_manager.cc:1929] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:36.777298 4357 log.cc:1199] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/d44e6e83e21f4c9485a4df3e40f8f72b
I20260501 13:58:36.778002 4357 ts_tablet_manager.cc:1950] T d44e6e83e21f4c9485a4df3e40f8f72b P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:36.778515 4061 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_06e2e16b792549e68d2e01ce3e3ce8b5" } new_table_name: "default.Table_995cfbea106b4656b4929310efb5017f" modify_external_catalogs: true
I20260501 13:58:36.781320 3802 tablet_service.cc:1511] Processing CreateTablet for tablet d5862fb4c6ca43feb61939b8698e728c (DEFAULT_TABLE table=default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:36.782349 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5862fb4c6ca43feb61939b8698e728c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:36.783915 4337 tablet_bootstrap.cc:492] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:36.784396 4337 ts_tablet_manager.cc:1403] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.000s
I20260501 13:58:36.785403 4380 leader_election.cc:290] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:36.786161 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02102952baaf40e0b897bc6dc04426da" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:36.787571 3822 raft_consensus.cc:3060] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:36.787582 4338 tablet_bootstrap.cc:492] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:36.788213 4338 ts_tablet_manager.cc:1403] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.102s user 0.012s sys 0.000s
I20260501 13:58:36.791565 4338 raft_consensus.cc:359] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.794152 4338 raft_consensus.cc:385] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.794528 4338 raft_consensus.cc:740] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.775528 4341 raft_consensus.cc:740] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.795894 4337 raft_consensus.cc:359] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.796553 4337 raft_consensus.cc:385] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.796835 4337 raft_consensus.cc:740] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.797149 4338 consensus_queue.cc:260] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.798707 4054 master_service.cc:954] ReplaceTablet: received request to replace tablet ba69864093504babbb00fb6170774778 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:36.800606 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1505e4bc749b48a88008d732881ff148" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:36.796172 4341 consensus_queue.cc:260] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.805991 4337 consensus_queue.cc:260] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.806838 3515 raft_consensus.cc:3060] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:36.810137 4380 raft_consensus.cc:493] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:36.813259 3515 raft_consensus.cc:2468] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:36.815325 4341 ts_tablet_manager.cc:1434] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20260501 13:58:36.825289 4357 tablet_replica.cc:333] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:36.827332 4357 raft_consensus.cc:2243] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:36.827939 4357 raft_consensus.cc:2272] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:36.826483 4337 ts_tablet_manager.cc:1434] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20260501 13:58:36.830907 4359 ts_tablet_manager.cc:1929] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:36.833539 4337 tablet_bootstrap.cc:492] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:36.838027 4359 log.cc:1199] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/cf6482b9e6994782b0b1ca4250af8e97
I20260501 13:58:36.838827 4337 tablet_bootstrap.cc:654] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.839607 4359 ts_tablet_manager.cc:1950] T cf6482b9e6994782b0b1ca4250af8e97 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:36.842417 4357 ts_tablet_manager.cc:1916] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:36.845383 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet cf6482b9e6994782b0b1ca4250af8e97 (table default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]) successfully deleted
I20260501 13:58:36.807953 4338 ts_tablet_manager.cc:1434] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260501 13:58:36.856436 4338 tablet_bootstrap.cc:492] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:36.860700 3822 raft_consensus.cc:2468] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:36.861617 4341 tablet_bootstrap.cc:492] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:36.865741 3448 leader_election.cc:304] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:36.801267 4382 leader_election.cc:290] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:36.869647 4338 tablet_bootstrap.cc:654] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.870414 4345 raft_consensus.cc:2804] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:36.870994 4345 raft_consensus.cc:697] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:36.871743 4345 consensus_queue.cc:237] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.880401 4052 catalog_manager.cc:5671] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:36.882401 4357 ts_tablet_manager.cc:1929] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:36.818100 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet d44e6e83e21f4c9485a4df3e40f8f72b (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:36.882880 4357 log.cc:1199] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/cf6482b9e6994782b0b1ca4250af8e97
W20260501 13:58:36.883184 4037 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.16.129:44335, user_credentials={real_user=slave}} blocked reactor thread for 65114us
I20260501 13:58:36.867178 4380 raft_consensus.cc:515] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.891016 4357 ts_tablet_manager.cc:1950] T cf6482b9e6994782b0b1ca4250af8e97 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:36.892254 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd4467544b3743a8acb44473f0b35203" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:36.892904 3674 raft_consensus.cc:2468] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:36.894130 3448 leader_election.cc:304] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:36.895077 4345 raft_consensus.cc:2804] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:36.895433 4345 raft_consensus.cc:493] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:36.895767 4345 raft_consensus.cc:3060] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:36.898353 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet cf6482b9e6994782b0b1ca4250af8e97 (table default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]) successfully deleted
I20260501 13:58:36.902683 4358 ts_tablet_manager.cc:1929] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:36.903092 4358 log.cc:1199] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/d44e6e83e21f4c9485a4df3e40f8f72b
W20260501 13:58:36.903499 4035 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 02102952baaf40e0b897bc6dc04426da consensus state has no leader
I20260501 13:58:36.903867 4358 ts_tablet_manager.cc:1950] T d44e6e83e21f4c9485a4df3e40f8f72b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:36.908255 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd4467544b3743a8acb44473f0b35203" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:36.908989 3822 raft_consensus.cc:2468] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:36.911854 4380 leader_election.cc:290] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:36.917048 4337 tablet_bootstrap.cc:492] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:36.917505 4337 ts_tablet_manager.cc:1403] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.087s user 0.022s sys 0.002s
I20260501 13:58:36.922672 4337 raft_consensus.cc:359] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:36.923323 4337 raft_consensus.cc:385] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.923609 4337 raft_consensus.cc:740] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.923936 4345 raft_consensus.cc:515] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.924271 4337 consensus_queue.cc:260] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:36.930371 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd4467544b3743a8acb44473f0b35203" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:36.931020 3674 raft_consensus.cc:3060] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:36.935381 4337 ts_tablet_manager.cc:1434] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260501 13:58:36.936555 4341 tablet_bootstrap.cc:654] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:36.939321 3674 raft_consensus.cc:2468] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:36.940732 3448 leader_election.cc:304] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:36.941620 4380 raft_consensus.cc:2804] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:36.942080 4380 raft_consensus.cc:697] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:36.942873 4380 consensus_queue.cc:237] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:36.943534 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd4467544b3743a8acb44473f0b35203" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:36.944082 3822 raft_consensus.cc:3060] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:36.951336 3822 raft_consensus.cc:2468] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:36.951717 4054 master_service.cc:954] ReplaceTablet: received request to replace tablet 7a626cb9dab0463eb9b23098a8adfd7d from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:36.953347 4345 leader_election.cc:290] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:36.984160 4341 tablet_bootstrap.cc:492] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:36.984970 4341 ts_tablet_manager.cc:1403] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.124s user 0.014s sys 0.000s
I20260501 13:58:36.989012 4050 catalog_manager.cc:3502] renamed table default.table_1f445634a0fc4676a2e7bf208b0ea1de [id=18ac75baaaf7485da21228e06a780ad2] in HMS: new name default.table_04a1387d0f0343998d94c1bb5bbff699
I20260501 13:58:36.987753 4341 raft_consensus.cc:359] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:36.989871 4341 raft_consensus.cc:385] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:36.990173 4341 raft_consensus.cc:740] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:36.991034 4341 consensus_queue.cc:260] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:37.017015 4341 ts_tablet_manager.cc:1434] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260501 13:58:37.025411 4054 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet 7a626cb9dab0463eb9b23098a8adfd7d
I20260501 13:58:37.027233 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a626cb9dab0463eb9b23098a8adfd7d with delete_type TABLET_DATA_DELETED (replaced by tablet 145974ff0af1418e839bf81b88d9f205) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:37.028013 4357 tablet_replica.cc:333] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:37.028811 4357 raft_consensus.cc:2243] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:37.029351 4357 raft_consensus.cc:2272] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:37.032693 4357 ts_tablet_manager.cc:1916] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:37.035745 4338 tablet_bootstrap.cc:492] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:37.036219 4338 ts_tablet_manager.cc:1403] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.181s user 0.009s sys 0.000s
I20260501 13:58:37.042665 4379 raft_consensus.cc:493] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:37.043165 4379 raft_consensus.cc:515] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:37.045023 4379 leader_election.cc:290] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
W20260501 13:58:37.049990 3756 leader_election.cc:343] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: d5862fb4c6ca43feb61939b8698e728c
I20260501 13:58:37.050035 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5862fb4c6ca43feb61939b8698e728c" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:37.050631 3515 raft_consensus.cc:2468] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:37.053335 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a626cb9dab0463eb9b23098a8adfd7d with delete_type TABLET_DATA_DELETED (replaced by tablet 145974ff0af1418e839bf81b88d9f205) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:37.054039 4359 tablet_replica.cc:333] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:37.054847 4359 raft_consensus.cc:2243] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:37.055377 4359 raft_consensus.cc:2272] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:37.055301 3757 leader_election.cc:304] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:37.056597 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet d44e6e83e21f4c9485a4df3e40f8f72b (table default.table_b07edfe247554299af96b76fee9a5b4e [id=a71004edce1a4d5298f01382612a99ee]) successfully deleted
I20260501 13:58:37.056951 4379 raft_consensus.cc:2804] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:37.057317 4379 raft_consensus.cc:493] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:37.057651 4379 raft_consensus.cc:3060] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:37.057705 4054 catalog_manager.cc:6741] ReplaceTablet: tablet 7a626cb9dab0463eb9b23098a8adfd7d deleted and replaced by tablet 145974ff0af1418e839bf81b88d9f205
I20260501 13:58:37.058844 4359 ts_tablet_manager.cc:1916] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:37.065225 4379 raft_consensus.cc:515] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:37.067000 4358 tablet_replica.cc:333] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:37.067808 4358 raft_consensus.cc:2243] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:37.068301 4358 raft_consensus.cc:2272] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:37.072299 4357 ts_tablet_manager.cc:1929] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:37.072750 4357 log.cc:1199] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/7a626cb9dab0463eb9b23098a8adfd7d
I20260501 13:58:37.074349 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5862fb4c6ca43feb61939b8698e728c" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:37.074928 3515 raft_consensus.cc:3060] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:37.078794 4338 raft_consensus.cc:359] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.079494 4338 raft_consensus.cc:385] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:37.079795 4338 raft_consensus.cc:740] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:37.081527 3515 raft_consensus.cc:2468] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:37.082008 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5862fb4c6ca43feb61939b8698e728c" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:37.082629 3757 leader_election.cc:304] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
W20260501 13:58:37.083415 3756 leader_election.cc:343] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:37.084015 4379 leader_election.cc:290] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:37.084637 4379 raft_consensus.cc:2804] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:37.085150 4379 raft_consensus.cc:697] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:37.085772 4338 consensus_queue.cc:260] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.087226 4379 consensus_queue.cc:237] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:37.088250 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a626cb9dab0463eb9b23098a8adfd7d with delete_type TABLET_DATA_DELETED (replaced by tablet 145974ff0af1418e839bf81b88d9f205) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:37.092689 4358 ts_tablet_manager.cc:1916] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:37.097980 4338 ts_tablet_manager.cc:1434] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.061s user 0.006s sys 0.002s
I20260501 13:58:37.100240 4357 ts_tablet_manager.cc:1950] T 7a626cb9dab0463eb9b23098a8adfd7d P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:37.103348 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 7a626cb9dab0463eb9b23098a8adfd7d (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
I20260501 13:58:37.118547 4338 tablet_bootstrap.cc:492] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:37.122880 4359 ts_tablet_manager.cc:1929] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:37.134677 4359 log.cc:1199] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/7a626cb9dab0463eb9b23098a8adfd7d
I20260501 13:58:37.135025 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 145974ff0af1418e839bf81b88d9f205 (DEFAULT_TABLE table=default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:37.135838 4359 ts_tablet_manager.cc:1950] T 7a626cb9dab0463eb9b23098a8adfd7d P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:37.136463 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145974ff0af1418e839bf81b88d9f205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:37.138375 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 7a626cb9dab0463eb9b23098a8adfd7d (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
I20260501 13:58:37.139334 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 145974ff0af1418e839bf81b88d9f205 (DEFAULT_TABLE table=default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:37.140266 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145974ff0af1418e839bf81b88d9f205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:37.140054 4052 catalog_manager.cc:5671] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9154033cea374a10a5225c53b8bcf122 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "9154033cea374a10a5225c53b8bcf122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: HEALTHY } } }
I20260501 13:58:37.141355 4052 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:37.142277 4052 catalog_manager.cc:5671] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9154033cea374a10a5225c53b8bcf122 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "9154033cea374a10a5225c53b8bcf122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: HEALTHY } } }
I20260501 13:58:37.152465 3493 tablet_service.cc:1511] Processing CreateTablet for tablet 145974ff0af1418e839bf81b88d9f205 (DEFAULT_TABLE table=default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:37.153517 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145974ff0af1418e839bf81b88d9f205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:37.160929 4338 tablet_bootstrap.cc:654] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:37.192881 4338 tablet_bootstrap.cc:492] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:37.193004 4341 tablet_bootstrap.cc:492] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:37.193440 4338 ts_tablet_manager.cc:1403] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.002s
I20260501 13:58:37.196322 4338 raft_consensus.cc:359] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.197084 4338 raft_consensus.cc:385] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:37.197335 4338 raft_consensus.cc:740] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:37.198596 4341 tablet_bootstrap.cc:654] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:37.209761 4337 tablet_bootstrap.cc:492] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:37.214675 4338 consensus_queue.cc:260] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.215662 4337 tablet_bootstrap.cc:654] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:37.215939 4341 tablet_bootstrap.cc:492] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:37.216346 4341 ts_tablet_manager.cc:1403] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20260501 13:58:37.223896 4341 raft_consensus.cc:359] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.224357 4338 ts_tablet_manager.cc:1434] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20260501 13:58:37.224529 4341 raft_consensus.cc:385] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:37.224786 4341 raft_consensus.cc:740] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:37.231712 4064 catalog_manager.cc:2174] created HMS catalog entry for table [id=095cec654bd6468da1aa74b795dc7942]
I20260501 13:58:37.231561 4341 consensus_queue.cc:260] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.232721 4337 tablet_bootstrap.cc:492] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:37.233140 4337 ts_tablet_manager.cc:1403] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.000s
I20260501 13:58:37.237097 4337 raft_consensus.cc:359] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.237697 4337 raft_consensus.cc:385] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:37.237936 4337 raft_consensus.cc:740] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:37.239907 4338 tablet_bootstrap.cc:492] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:37.242944 4341 ts_tablet_manager.cc:1434] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260501 13:58:37.245889 4337 consensus_queue.cc:260] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.253144 4337 ts_tablet_manager.cc:1434] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260501 13:58:37.259330 4338 tablet_bootstrap.cc:654] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:37.269793 4380 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:37.270390 4380 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:37.272562 4064 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 50) took 1477 ms. Trace:
W20260501 13:58:37.273754 3608 consensus_peers.cc:597] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 -> Peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Couldn't send request to peer 5c239ccbdc554619a237d78e958ffbc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a626cb9dab0463eb9b23098a8adfd7d. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:58:37.273978 4380 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:37.277088 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:58:37.277772 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:58:37.278138 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:37.279162 4380 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:37.272908 4064 rpcz_store.cc:276] 0501 13:58:35.795239 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:35.795418 (+ 179us) service_pool.cc:224] Handling call
0501 13:58:36.559357 (+763939us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:36.561164 (+ 1807us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:37.231670 (+670506us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:37.271854 (+ 40184us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:37.272199 (+ 345us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:37.272484 (+ 285us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:37.234334 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:37.234823 (+ 489us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:37.234849 (+ 26us) write_op.cc:435] Acquired schema lock
0501 13:58:37.234864 (+ 15us) tablet.cc:662] Decoding operations
0501 13:58:37.235153 (+ 289us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:37.235198 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:37.235224 (+ 26us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:37.235447 (+ 223us) tablet.cc:701] Row locks acquired
0501 13:58:37.235462 (+ 15us) write_op.cc:260] PREPARE: finished
0501 13:58:37.235604 (+ 142us) write_op.cc:270] Start()
0501 13:58:37.235693 (+ 89us) write_op.cc:276] Timestamp: P: 1777643917235553 usec, L: 0
0501 13:58:37.235706 (+ 13us) op_driver.cc:348] REPLICATION: starting
0501 13:58:37.236240 (+ 534us) log.cc:844] Serialized 433 byte log entry
0501 13:58:37.246179 (+ 9939us) op_driver.cc:464] REPLICATION: finished
0501 13:58:37.246633 (+ 454us) write_op.cc:301] APPLY: starting
0501 13:58:37.246694 (+ 61us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:37.246793 (+ 99us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:37.246878 (+ 85us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:37.247255 (+ 377us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:37.247388 (+ 133us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:37.247432 (+ 44us) write_op.cc:312] APPLY: finished
0501 13:58:37.247641 (+ 209us) log.cc:844] Serialized 29 byte log entry
0501 13:58:37.247774 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:37.247845 (+ 71us) write_op.cc:454] Released schema lock
0501 13:58:37.247875 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:37.248077 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:37.248099 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":426804,"Hive Metastore.run_cpu_time_us":627,"Hive Metastore.run_wall_time_us":242885,"mutex_wait_us":3370,"child_traces":[["op",{"apply.queue_time_us":293,"apply.run_cpu_time_us":1419,"apply.run_wall_time_us":1419,"num_ops":2,"prepare.queue_time_us":115,"prepare.run_cpu_time_us":2253,"prepare.run_wall_time_us":2252,"raft.queue_time_us":2690,"raft.run_cpu_time_us":662,"raft.run_wall_time_us":663,"replication_time_us":10370}]]}
I20260501 13:58:37.289505 4382 raft_consensus.cc:493] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:37.290061 4382 raft_consensus.cc:515] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.290323 4064 catalog_manager.cc:5671] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } }
I20260501 13:58:37.292377 4382 leader_election.cc:290] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:37.294405 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "528c2d886bab4375b79f969343d9f46b" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:37.295146 3822 raft_consensus.cc:2468] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:37.296715 3608 leader_election.cc:304] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:37.301829 4382 raft_consensus.cc:2804] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:37.302213 4382 raft_consensus.cc:493] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:37.302703 4382 raft_consensus.cc:3060] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:37.305930 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "528c2d886bab4375b79f969343d9f46b" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:37.306521 3515 raft_consensus.cc:2468] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:37.307176 4380 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:37.307595 4380 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:37.309417 4380 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:37.313901 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 13:58:37.315793 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:37.316206 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:37.316866 4380 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:37.310288 4382 raft_consensus.cc:515] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.324491 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "528c2d886bab4375b79f969343d9f46b" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:37.325153 3822 raft_consensus.cc:3060] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:37.343163 3822 raft_consensus.cc:2468] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:37.344318 3608 leader_election.cc:304] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:37.349416 4338 tablet_bootstrap.cc:492] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:37.349938 4338 ts_tablet_manager.cc:1403] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.110s user 0.010s sys 0.000s
I20260501 13:58:37.360105 4338 raft_consensus.cc:359] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:37.360870 4338 raft_consensus.cc:385] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:37.361146 4338 raft_consensus.cc:740] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:37.368882 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "528c2d886bab4375b79f969343d9f46b" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:37.369520 3515 raft_consensus.cc:3060] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:37.376150 4054 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 145974ff0af1418e839bf81b88d9f205 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]). Expected version 1 got 0
I20260501 13:58:37.377354 3515 raft_consensus.cc:2468] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:37.379632 4382 leader_election.cc:290] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:37.380946 4382 raft_consensus.cc:2804] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:37.382997 4382 raft_consensus.cc:697] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:37.385426 4064 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 145974ff0af1418e839bf81b88d9f205 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]). Expected version 1 got 0
I20260501 13:58:37.388638 4382 consensus_queue.cc:237] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.395104 4338 consensus_queue.cc:260] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:37.412390 4338 ts_tablet_manager.cc:1434] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.062s user 0.005s sys 0.004s
I20260501 13:58:37.418006 4338 tablet_bootstrap.cc:492] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:37.222220 4052 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:37.457583 4052 master_service.cc:954] ReplaceTablet: received request to replace tablet 528c2d886bab4375b79f969343d9f46b from {username='slave'} at 127.0.0.1:35438
W20260501 13:58:37.451210 4054 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 145974ff0af1418e839bf81b88d9f205 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 145974ff0af1418e839bf81b88d9f205 consensus state has no leader
I20260501 13:58:37.462006 4338 tablet_bootstrap.cc:654] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:37.480973 4338 tablet_bootstrap.cc:492] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:37.481462 4338 ts_tablet_manager.cc:1403] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.064s user 0.010s sys 0.000s
I20260501 13:58:37.484400 4358 ts_tablet_manager.cc:1929] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260501 13:58:37.513062 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:37.484812 4338 raft_consensus.cc:359] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.514961 4338 raft_consensus.cc:385] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:37.515231 4358 log.cc:1199] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/cf6482b9e6994782b0b1ca4250af8e97
I20260501 13:58:37.515307 4338 raft_consensus.cc:740] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:37.522781 4338 consensus_queue.cc:260] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.524689 4338 ts_tablet_manager.cc:1434] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.043s user 0.008s sys 0.000s
I20260501 13:58:37.529753 4382 raft_consensus.cc:493] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:37.530182 4382 raft_consensus.cc:515] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.532006 4382 leader_election.cc:290] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:37.533442 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02102952baaf40e0b897bc6dc04426da" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:37.536124 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02102952baaf40e0b897bc6dc04426da" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:37.536881 3822 raft_consensus.cc:2393] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in current term 1: Already voted for candidate 9154033cea374a10a5225c53b8bcf122 in this term.
I20260501 13:58:37.538684 3608 leader_election.cc:304] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:37.539959 4382 raft_consensus.cc:3060] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:37.547919 4382 raft_consensus.cc:2749] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260501 13:58:37.549093 4064 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
I20260501 13:58:37.549383 4358 ts_tablet_manager.cc:1950] T cf6482b9e6994782b0b1ca4250af8e97 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:37.560896 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet cf6482b9e6994782b0b1ca4250af8e97 (table default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]) successfully deleted
I20260501 13:58:37.562268 4052 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet 528c2d886bab4375b79f969343d9f46b
I20260501 13:58:37.563966 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 528c2d886bab4375b79f969343d9f46b with delete_type TABLET_DATA_DELETED (replaced by tablet df6234730e704c77a3a85a407f0e1cff) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:37.565739 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 528c2d886bab4375b79f969343d9f46b with delete_type TABLET_DATA_DELETED (replaced by tablet df6234730e704c77a3a85a407f0e1cff) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:37.566426 4359 tablet_replica.cc:333] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:37.566509 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet 528c2d886bab4375b79f969343d9f46b with delete_type TABLET_DATA_DELETED (replaced by tablet df6234730e704c77a3a85a407f0e1cff) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:37.567212 4357 tablet_replica.cc:333] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:37.567229 4359 raft_consensus.cc:2243] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:37.567696 4359 raft_consensus.cc:2272] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:37.568179 4357 raft_consensus.cc:2243] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:37.568709 4357 raft_consensus.cc:2272] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:37.569621 4359 ts_tablet_manager.cc:1916] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:37.569998 4358 tablet_replica.cc:333] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:37.570660 4357 ts_tablet_manager.cc:1916] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:37.570850 4358 raft_consensus.cc:2243] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:37.571805 4358 raft_consensus.cc:2272] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:37.574636 4052 catalog_manager.cc:6741] ReplaceTablet: tablet 528c2d886bab4375b79f969343d9f46b deleted and replaced by tablet df6234730e704c77a3a85a407f0e1cff
I20260501 13:58:37.576457 4052 master_service.cc:954] ReplaceTablet: received request to replace tablet a9ecb5b51ae04f32a4f07f828d563fa4 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:37.582355 4052 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_8be6ed54e88344f7b4be783af8fa48da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:37.591152 4359 ts_tablet_manager.cc:1929] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:37.592726 4358 ts_tablet_manager.cc:1916] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:37.598676 4359 log.cc:1199] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/528c2d886bab4375b79f969343d9f46b
I20260501 13:58:37.599339 4379 raft_consensus.cc:493] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:37.599798 4379 raft_consensus.cc:515] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.599573 4359 ts_tablet_manager.cc:1950] T 528c2d886bab4375b79f969343d9f46b P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:37.601724 4379 leader_election.cc:290] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:37.605363 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145974ff0af1418e839bf81b88d9f205" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:37.606009 3674 raft_consensus.cc:2468] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:37.607034 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145974ff0af1418e839bf81b88d9f205" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:37.607503 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 528c2d886bab4375b79f969343d9f46b (table default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]) successfully deleted
I20260501 13:58:37.607604 3515 raft_consensus.cc:2468] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:37.609547 3757 leader_election.cc:304] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:37.611915 4379 raft_consensus.cc:2804] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:37.612257 4379 raft_consensus.cc:493] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:37.612543 4379 raft_consensus.cc:3060] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:37.620059 4357 ts_tablet_manager.cc:1929] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:37.620530 4357 log.cc:1199] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/528c2d886bab4375b79f969343d9f46b
I20260501 13:58:37.621212 4357 ts_tablet_manager.cc:1950] T 528c2d886bab4375b79f969343d9f46b P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:37.623991 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 528c2d886bab4375b79f969343d9f46b (table default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]) successfully deleted
I20260501 13:58:37.624555 3802 tablet_service.cc:1511] Processing CreateTablet for tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:37.624895 3654 tablet_service.cc:1511] Processing CreateTablet for tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:37.625571 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:37.625710 3493 tablet_service.cc:1511] Processing CreateTablet for tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:37.626195 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:37.626654 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:37.639912 4064 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 145974ff0af1418e839bf81b88d9f205 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]). Expected version 1 got 0
I20260501 13:58:37.638495 4379 raft_consensus.cc:515] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.647135 4379 leader_election.cc:290] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:37.648773 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145974ff0af1418e839bf81b88d9f205" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:37.649358 3515 raft_consensus.cc:3060] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:37.655023 4338 tablet_bootstrap.cc:492] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:37.658509 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145974ff0af1418e839bf81b88d9f205" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:37.659327 3674 raft_consensus.cc:3060] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:37.673372 4054 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 145974ff0af1418e839bf81b88d9f205 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]). Expected version 1 got 0
I20260501 13:58:37.676086 4341 tablet_bootstrap.cc:492] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:37.668268 4338 tablet_bootstrap.cc:654] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:37.681706 4341 tablet_bootstrap.cc:654] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:37.692282 3515 raft_consensus.cc:2468] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:37.698053 3674 raft_consensus.cc:2468] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:37.702713 4066 catalog_manager.cc:2174] created HMS catalog entry for table [id=5b3f55db8abb4fa7a9b3abf286620942]
I20260501 13:58:37.705476 3757 leader_election.cc:304] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:37.706634 4379 raft_consensus.cc:2804] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:37.719517 4066 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 51) took 1801 ms. Trace:
I20260501 13:58:37.719907 4066 rpcz_store.cc:276] 0501 13:58:35.918314 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:35.918521 (+ 207us) service_pool.cc:224] Handling call
0501 13:58:36.563155 (+644634us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:36.564823 (+ 1668us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:37.702662 (+1137839us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:37.718836 (+ 16174us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:37.719148 (+ 312us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:37.719434 (+ 286us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:37.704647 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:37.705131 (+ 484us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:37.705160 (+ 29us) write_op.cc:435] Acquired schema lock
0501 13:58:37.705176 (+ 16us) tablet.cc:662] Decoding operations
0501 13:58:37.705478 (+ 302us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:37.705533 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:37.705563 (+ 30us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:37.705807 (+ 244us) tablet.cc:701] Row locks acquired
0501 13:58:37.705825 (+ 18us) write_op.cc:260] PREPARE: finished
0501 13:58:37.705958 (+ 133us) write_op.cc:270] Start()
0501 13:58:37.706055 (+ 97us) write_op.cc:276] Timestamp: P: 1777643917705914 usec, L: 0
0501 13:58:37.706071 (+ 16us) op_driver.cc:348] REPLICATION: starting
0501 13:58:37.706904 (+ 833us) log.cc:844] Serialized 433 byte log entry
0501 13:58:37.715177 (+ 8273us) op_driver.cc:464] REPLICATION: finished
0501 13:58:37.716476 (+ 1299us) write_op.cc:301] APPLY: starting
0501 13:58:37.716562 (+ 86us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:37.716668 (+ 106us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:37.716758 (+ 90us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:37.717134 (+ 376us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:37.717270 (+ 136us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:37.717309 (+ 39us) write_op.cc:312] APPLY: finished
0501 13:58:37.717486 (+ 177us) log.cc:844] Serialized 29 byte log entry
0501 13:58:37.717639 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:37.717706 (+ 67us) write_op.cc:454] Released schema lock
0501 13:58:37.717755 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:37.717965 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:37.717986 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":666162,"Hive Metastore.run_cpu_time_us":634,"Hive Metastore.run_wall_time_us":469730,"child_traces":[["op",{"apply.queue_time_us":1124,"apply.run_cpu_time_us":1473,"apply.run_wall_time_us":1473,"num_ops":2,"prepare.queue_time_us":135,"prepare.run_cpu_time_us":2368,"prepare.run_wall_time_us":2582,"raft.queue_time_us":175,"raft.run_cpu_time_us":655,"raft.run_wall_time_us":654,"replication_time_us":9001}]]}
I20260501 13:58:37.800050 4066 master_service.cc:954] ReplaceTablet: received request to replace tablet fd4467544b3743a8acb44473f0b35203 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:37.803596 4064 master_service.cc:954] ReplaceTablet: received request to replace tablet ba69864093504babbb00fb6170774778 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:37.806483 3802 tablet_service.cc:1511] Processing CreateTablet for tablet df6234730e704c77a3a85a407f0e1cff (DEFAULT_TABLE table=default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:37.807515 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6234730e704c77a3a85a407f0e1cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:37.761018 4358 ts_tablet_manager.cc:1929] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:37.804783 3654 tablet_service.cc:1511] Processing CreateTablet for tablet df6234730e704c77a3a85a407f0e1cff (DEFAULT_TABLE table=default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:37.814682 4358 log.cc:1199] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/7a626cb9dab0463eb9b23098a8adfd7d
I20260501 13:58:37.815186 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6234730e704c77a3a85a407f0e1cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:37.815446 4358 ts_tablet_manager.cc:1950] T 7a626cb9dab0463eb9b23098a8adfd7d P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:37.817521 4379 raft_consensus.cc:697] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:37.818367 4379 consensus_queue.cc:237] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
W20260501 13:58:37.822119 4037 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.189:44255, user_credentials={real_user=slave}} blocked reactor thread for 138575us
I20260501 13:58:37.824734 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 7a626cb9dab0463eb9b23098a8adfd7d (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
I20260501 13:58:37.825757 4358 tablet_replica.cc:333] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:37.826644 4358 raft_consensus.cc:2243] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:37.827425 4358 pending_rounds.cc:70] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: Trying to abort 1 pending ops.
I20260501 13:58:37.827656 4358 pending_rounds.cc:77] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229485723140096 op_type: NO_OP noop_request { }
I20260501 13:58:37.830583 4358 raft_consensus.cc:2889] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:37.830983 4358 raft_consensus.cc:2272] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:37.835944 4338 tablet_bootstrap.cc:492] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:37.836154 4341 tablet_bootstrap.cc:492] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:37.836673 4341 ts_tablet_manager.cc:1403] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.161s user 0.016s sys 0.014s
I20260501 13:58:37.839112 4341 raft_consensus.cc:359] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.839581 4358 ts_tablet_manager.cc:1916] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:37.840641 3492 tablet_service.cc:1511] Processing CreateTablet for tablet df6234730e704c77a3a85a407f0e1cff (DEFAULT_TABLE table=default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:37.844589 3492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6234730e704c77a3a85a407f0e1cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:37.842342 4338 ts_tablet_manager.cc:1403] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.188s user 0.009s sys 0.004s
I20260501 13:58:37.839730 4341 raft_consensus.cc:385] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:37.862934 4341 raft_consensus.cc:740] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:37.863775 4341 consensus_queue.cc:260] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.865718 4433 tablet_bootstrap.cc:492] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:37.867848 4358 ts_tablet_manager.cc:1929] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:37.868256 4358 log.cc:1199] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/528c2d886bab4375b79f969343d9f46b
I20260501 13:58:37.869076 4358 ts_tablet_manager.cc:1950] T 528c2d886bab4375b79f969343d9f46b P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:37.871636 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 528c2d886bab4375b79f969343d9f46b (table default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]) successfully deleted
I20260501 13:58:37.876951 4341 ts_tablet_manager.cc:1434] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20260501 13:58:37.877882 4341 tablet_bootstrap.cc:492] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:37.881395 4382 raft_consensus.cc:493] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:37.881886 4382 raft_consensus.cc:515] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:37.882620 4421 consensus_queue.cc:1048] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:37.883808 4382 leader_election.cc:290] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:37.885516 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5862fb4c6ca43feb61939b8698e728c" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:37.886423 3515 raft_consensus.cc:2393] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in current term 1: Already voted for candidate 5c239ccbdc554619a237d78e958ffbc2 in this term.
I20260501 13:58:37.893476 4341 tablet_bootstrap.cc:654] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:37.897811 4066 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet fd4467544b3743a8acb44473f0b35203
I20260501 13:58:37.899664 4066 catalog_manager.cc:6741] ReplaceTablet: tablet fd4467544b3743a8acb44473f0b35203 deleted and replaced by tablet 533eb8337c414aea98ea95d7bf507b1e
I20260501 13:58:37.900804 3495 tablet_service.cc:1558] Processing DeleteTablet for tablet fd4467544b3743a8acb44473f0b35203 with delete_type TABLET_DATA_DELETED (replaced by tablet 533eb8337c414aea98ea95d7bf507b1e) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:37.900966 4433 tablet_bootstrap.cc:654] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:37.901539 4357 tablet_replica.cc:333] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:37.902297 4357 raft_consensus.cc:2243] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:37.902968 4357 pending_rounds.cc:70] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: Trying to abort 1 pending ops.
I20260501 13:58:37.902693 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet fd4467544b3743a8acb44473f0b35203 with delete_type TABLET_DATA_DELETED (replaced by tablet 533eb8337c414aea98ea95d7bf507b1e) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:37.903193 4357 pending_rounds.cc:77] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229483809763328 op_type: NO_OP noop_request { }
I20260501 13:58:37.903560 4357 raft_consensus.cc:2889] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:37.903944 4357 raft_consensus.cc:2272] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:37.903904 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet fd4467544b3743a8acb44473f0b35203 with delete_type TABLET_DATA_DELETED (replaced by tablet 533eb8337c414aea98ea95d7bf507b1e) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:37.904663 4359 tablet_replica.cc:333] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:37.904887 4358 tablet_replica.cc:333] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:37.905496 4359 raft_consensus.cc:2243] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:37.905717 4358 raft_consensus.cc:2243] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:37.905984 4359 raft_consensus.cc:2272] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:37.906234 4358 raft_consensus.cc:2272] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:37.907994 4359 ts_tablet_manager.cc:1916] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:37.908425 4358 ts_tablet_manager.cc:1916] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:37.915179 4357 ts_tablet_manager.cc:1916] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:37.938170 4421 consensus_queue.cc:1048] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:37.945365 4064 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_59dc65b11ade4ee3acb91116f7a60585"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:37.945940 4370 consensus_queue.cc:1048] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:37.959070 4359 ts_tablet_manager.cc:1929] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:37.960431 4359 log.cc:1199] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/fd4467544b3743a8acb44473f0b35203
I20260501 13:58:37.961230 4359 ts_tablet_manager.cc:1950] T fd4467544b3743a8acb44473f0b35203 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:37.962280 4358 ts_tablet_manager.cc:1929] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:37.962694 4358 log.cc:1199] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/fd4467544b3743a8acb44473f0b35203
I20260501 13:58:37.963299 4358 ts_tablet_manager.cc:1950] T fd4467544b3743a8acb44473f0b35203 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:37.964636 4433 tablet_bootstrap.cc:492] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:37.965150 4433 ts_tablet_manager.cc:1403] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.100s user 0.007s sys 0.004s
I20260501 13:58:37.966269 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet fd4467544b3743a8acb44473f0b35203 (table default.table_f52f85ad11fb4023adead2ab9c5eb694 [id=53fa808a27054cdf834732393dc08838]) successfully deleted
I20260501 13:58:37.969697 4428 raft_consensus.cc:493] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:37.970161 4428 raft_consensus.cc:515] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:37.972045 4428 leader_election.cc:290] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:37.974175 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfea0ce6458e4d5a8e9a5dbe87fa28c3" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:37.979189 4357 ts_tablet_manager.cc:1929] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:37.979669 4357 log.cc:1199] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/fd4467544b3743a8acb44473f0b35203
I20260501 13:58:37.980422 4357 ts_tablet_manager.cc:1950] T fd4467544b3743a8acb44473f0b35203 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:37.979641 4066 catalog_manager.cc:5671] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:37.981352 4066 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 145974ff0af1418e839bf81b88d9f205 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]). Expected version 1 got 0
I20260501 13:58:37.983451 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5862fb4c6ca43feb61939b8698e728c" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:37.989657 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet fd4467544b3743a8acb44473f0b35203 (table default.table_f52f85ad11fb4023adead2ab9c5eb694 [id=53fa808a27054cdf834732393dc08838]) successfully deleted
I20260501 13:58:38.000373 4338 raft_consensus.cc:359] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.001315 4341 tablet_bootstrap.cc:492] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:38.001758 4341 ts_tablet_manager.cc:1403] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.124s user 0.008s sys 0.000s
I20260501 13:58:38.017468 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet fd4467544b3743a8acb44473f0b35203 (table default.table_f52f85ad11fb4023adead2ab9c5eb694 [id=53fa808a27054cdf834732393dc08838]) successfully deleted
I20260501 13:58:38.017958 4433 raft_consensus.cc:359] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.018680 4433 raft_consensus.cc:385] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.018963 4433 raft_consensus.cc:740] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:38.019654 4433 consensus_queue.cc:260] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.020478 4338 raft_consensus.cc:385] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.021068 4338 raft_consensus.cc:740] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:38.022691 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfea0ce6458e4d5a8e9a5dbe87fa28c3" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:38.023306 3513 raft_consensus.cc:2468] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:38.030089 4341 raft_consensus.cc:359] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.030792 4341 raft_consensus.cc:385] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.031046 4341 raft_consensus.cc:740] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
W20260501 13:58:38.036803 4055 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c0462a887c864245890f76dea029e11b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:38.046586 4038 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 145974ff0af1418e839bf81b88d9f205 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 145974ff0af1418e839bf81b88d9f205 consensus state has no leader
W20260501 13:58:38.049084 4059 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_124cdf87baa148fbbc6e5f06d1efcda1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:38.052742 4056 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35438 (request call id 44) took 3339 ms. Trace:
I20260501 13:58:38.053121 4056 rpcz_store.cc:276] 0501 13:58:34.713313 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:34.713453 (+ 140us) service_pool.cc:224] Handling call
0501 13:58:36.141707 (+1428254us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:38.052667 (+1910960us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":27930,"Hive Metastore.run_cpu_time_us":417,"Hive Metastore.run_wall_time_us":297270,"mutex_wait_us":24141}
I20260501 13:58:38.056392 4054 master_service.cc:954] ReplaceTablet: received request to replace tablet 02102952baaf40e0b897bc6dc04426da from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:38.022780 3608 leader_election.cc:304] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122
W20260501 13:58:38.080580 3608 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:38685, user_credentials={real_user=slave}} blocked reactor thread for 57893.5us
I20260501 13:58:38.081454 4382 raft_consensus.cc:3060] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:38.093022 4338 consensus_queue.cc:260] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.093971 3672 raft_consensus.cc:2468] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:38.095747 3756 leader_election.cc:304] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:38.096971 4379 raft_consensus.cc:2804] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:38.097371 4379 raft_consensus.cc:493] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:38.097698 4379 raft_consensus.cc:3060] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:38.102015 4382 raft_consensus.cc:2749] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:38.104339 4451 tablet.cc:1724] T 545ced72d1cd47faa827d7b00df372b7 P 5c239ccbdc554619a237d78e958ffbc2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:38.106294 4452 tablet.cc:1724] T 545ced72d1cd47faa827d7b00df372b7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:38.113600 4056 master_service.cc:954] ReplaceTablet: received request to replace tablet 145974ff0af1418e839bf81b88d9f205 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:38.119067 4433 ts_tablet_manager.cc:1434] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.154s user 0.006s sys 0.000s
I20260501 13:58:38.119266 3672 raft_consensus.cc:1275] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Refusing update from remote peer 9154033cea374a10a5225c53b8bcf122: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:58:38.120129 4433 tablet_bootstrap.cc:492] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:38.121475 4370 consensus_queue.cc:1048] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:38.138571 4341 consensus_queue.cc:260] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.149582 4338 ts_tablet_manager.cc:1434] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.301s user 0.005s sys 0.001s
I20260501 13:58:38.150668 4338 tablet_bootstrap.cc:492] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:38.160328 4341 ts_tablet_manager.cc:1434] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.158s user 0.007s sys 0.000s
I20260501 13:58:38.165261 4054 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet 02102952baaf40e0b897bc6dc04426da
I20260501 13:58:38.169121 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 02102952baaf40e0b897bc6dc04426da with delete_type TABLET_DATA_DELETED (replaced by tablet 41ac3120d65440829fe929b76cc41d43) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:38.169888 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 02102952baaf40e0b897bc6dc04426da with delete_type TABLET_DATA_DELETED (replaced by tablet 41ac3120d65440829fe929b76cc41d43) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:38.170617 4358 tablet_replica.cc:333] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:38.171780 4359 tablet_replica.cc:333] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:38.172535 4359 raft_consensus.cc:2243] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:38.173033 4359 pending_rounds.cc:70] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: Trying to abort 2 pending ops.
I20260501 13:58:38.173311 4359 pending_rounds.cc:77] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7281229487856435200 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "02102952baaf40e0b897bc6dc04426da" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.table_4db61a7562964779afbc5ff93b00a5f5" dest_uuid: "9154033cea374a10a5225c53b8bcf122" new_extra_config { } }
I20260501 13:58:38.174748 4359 pending_rounds.cc:77] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229483515162624 op_type: NO_OP noop_request { }
I20260501 13:58:38.175158 4359 raft_consensus.cc:2889] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:38.175549 4359 raft_consensus.cc:2272] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:38.178085 4359 ts_tablet_manager.cc:1916] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:38.189134 4054 catalog_manager.cc:6741] ReplaceTablet: tablet 02102952baaf40e0b897bc6dc04426da deleted and replaced by tablet 41ac3120d65440829fe929b76cc41d43
I20260501 13:58:38.189527 4358 raft_consensus.cc:2243] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:38.190973 4358 pending_rounds.cc:70] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: Trying to abort 2 pending ops.
I20260501 13:58:38.191257 4358 pending_rounds.cc:77] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7281229487856435200 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "02102952baaf40e0b897bc6dc04426da" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.table_4db61a7562964779afbc5ff93b00a5f5" dest_uuid: "9154033cea374a10a5225c53b8bcf122" new_extra_config { } }
I20260501 13:58:38.192624 4358 pending_rounds.cc:77] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229483515162624 op_type: NO_OP noop_request { }
I20260501 13:58:38.193049 4358 raft_consensus.cc:2889] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:38.193416 4358 raft_consensus.cc:2272] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:38.194021 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet 02102952baaf40e0b897bc6dc04426da with delete_type TABLET_DATA_DELETED (replaced by tablet 41ac3120d65440829fe929b76cc41d43) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:38.195065 4357 tablet_replica.cc:333] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:38.195760 4357 raft_consensus.cc:2243] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:38.196290 4357 pending_rounds.cc:70] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: Trying to abort 2 pending ops.
I20260501 13:58:38.196501 4357 pending_rounds.cc:77] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7281229487856435200 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "02102952baaf40e0b897bc6dc04426da" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.table_4db61a7562964779afbc5ff93b00a5f5" dest_uuid: "9154033cea374a10a5225c53b8bcf122" new_extra_config { } }
W20260501 13:58:38.197528 4357 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52150: Aborted: Op aborted
W20260501 13:58:38.199087 4037 catalog_manager.cc:5081] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): alter failed for tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]): Aborted: Op aborted
I20260501 13:58:38.198413 4357 pending_rounds.cc:77] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229483515162624 op_type: NO_OP noop_request { }
I20260501 13:58:38.200146 4357 raft_consensus.cc:2889] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:38.200482 4357 raft_consensus.cc:2272] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:38.202073 4379 raft_consensus.cc:515] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.202857 4357 ts_tablet_manager.cc:1916] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:38.203652 4443 consensus_queue.cc:1048] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:58:38.209059 4457 tablet.cc:1724] T 545ced72d1cd47faa827d7b00df372b7 P 9154033cea374a10a5225c53b8bcf122: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:38.217949 4358 ts_tablet_manager.cc:1916] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:38.228592 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfea0ce6458e4d5a8e9a5dbe87fa28c3" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:38.229317 3674 raft_consensus.cc:3060] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:38.238245 4379 leader_election.cc:290] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:38.243301 4433 tablet_bootstrap.cc:654] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:38.246371 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfea0ce6458e4d5a8e9a5dbe87fa28c3" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:38.246980 3515 raft_consensus.cc:3060] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:38.253120 3515 raft_consensus.cc:2468] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:38.259030 3757 leader_election.cc:304] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:38.260056 4443 raft_consensus.cc:2804] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:38.260514 4443 raft_consensus.cc:697] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:38.243371 3674 raft_consensus.cc:2468] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:38.261281 4443 consensus_queue.cc:237] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.265916 4428 consensus_queue.cc:1048] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:38.277827 4359 ts_tablet_manager.cc:1929] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:38.278276 4359 log.cc:1199] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/02102952baaf40e0b897bc6dc04426da
I20260501 13:58:38.288343 4433 tablet_bootstrap.cc:492] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:38.288836 4433 ts_tablet_manager.cc:1403] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.169s user 0.005s sys 0.004s
I20260501 13:58:38.304462 4357 ts_tablet_manager.cc:1929] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:38.304879 4357 log.cc:1199] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/02102952baaf40e0b897bc6dc04426da
I20260501 13:58:38.305603 4357 ts_tablet_manager.cc:1950] T 02102952baaf40e0b897bc6dc04426da P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:38.310417 4433 raft_consensus.cc:359] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.311209 4433 raft_consensus.cc:385] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.311497 4433 raft_consensus.cc:740] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:38.313864 4358 ts_tablet_manager.cc:1929] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:38.314491 4358 log.cc:1199] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/02102952baaf40e0b897bc6dc04426da
I20260501 13:58:38.316040 4056 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet 145974ff0af1418e839bf81b88d9f205
I20260501 13:58:38.317488 4056 catalog_manager.cc:6741] ReplaceTablet: tablet 145974ff0af1418e839bf81b88d9f205 deleted and replaced by tablet d6c11a5794f64e69a3f467800a7cd2ae
I20260501 13:58:38.321193 4055 catalog_manager.cc:2174] created HMS catalog entry for table [id=52fc296b289947f1be9e9f74d320675c]
I20260501 13:58:38.323290 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet 145974ff0af1418e839bf81b88d9f205 with delete_type TABLET_DATA_DELETED (replaced by tablet d6c11a5794f64e69a3f467800a7cd2ae) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:38.324455 4359 ts_tablet_manager.cc:1950] T 02102952baaf40e0b897bc6dc04426da P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:38.327387 4358 ts_tablet_manager.cc:1950] T 02102952baaf40e0b897bc6dc04426da P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:38.332120 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 145974ff0af1418e839bf81b88d9f205 with delete_type TABLET_DATA_DELETED (replaced by tablet d6c11a5794f64e69a3f467800a7cd2ae) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:38.333245 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 145974ff0af1418e839bf81b88d9f205 with delete_type TABLET_DATA_DELETED (replaced by tablet d6c11a5794f64e69a3f467800a7cd2ae) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:38.336567 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:38.337245 4358 tablet_replica.cc:333] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:38.338089 4358 raft_consensus.cc:2243] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:38.338268 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:38.338611 4358 raft_consensus.cc:2272] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:38.340401 4358 ts_tablet_manager.cc:1916] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:38.335727 4433 consensus_queue.cc:260] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.347627 4357 tablet_replica.cc:333] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:38.350970 4433 ts_tablet_manager.cc:1434] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.062s user 0.008s sys 0.000s
I20260501 13:58:38.352573 4357 raft_consensus.cc:2243] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:38.353085 4357 raft_consensus.cc:2272] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:38.355155 4358 ts_tablet_manager.cc:1929] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:38.355258 4055 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 52) took 2176 ms. Trace:
I20260501 13:58:38.355621 4358 log.cc:1199] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/145974ff0af1418e839bf81b88d9f205
I20260501 13:58:38.356330 4358 ts_tablet_manager.cc:1950] T 145974ff0af1418e839bf81b88d9f205 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:38.359010 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 145974ff0af1418e839bf81b88d9f205 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
I20260501 13:58:38.355608 4055 rpcz_store.cc:276] 0501 13:58:36.178909 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:36.179071 (+ 162us) service_pool.cc:224] Handling call
0501 13:58:38.037476 (+1858405us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:38.039814 (+ 2338us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:38.321145 (+281331us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:38.354237 (+ 33092us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:38.354826 (+ 589us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:38.355184 (+ 358us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:38.326793 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:38.327187 (+ 394us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:38.327212 (+ 25us) write_op.cc:435] Acquired schema lock
0501 13:58:38.327225 (+ 13us) tablet.cc:662] Decoding operations
0501 13:58:38.327489 (+ 264us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:38.327527 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:38.327550 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:38.327821 (+ 271us) tablet.cc:701] Row locks acquired
0501 13:58:38.327834 (+ 13us) write_op.cc:260] PREPARE: finished
0501 13:58:38.327946 (+ 112us) write_op.cc:270] Start()
0501 13:58:38.328026 (+ 80us) write_op.cc:276] Timestamp: P: 1777643918327895 usec, L: 0
0501 13:58:38.328041 (+ 15us) op_driver.cc:348] REPLICATION: starting
0501 13:58:38.328499 (+ 458us) log.cc:844] Serialized 433 byte log entry
0501 13:58:38.352060 (+ 23561us) op_driver.cc:464] REPLICATION: finished
0501 13:58:38.352305 (+ 245us) write_op.cc:301] APPLY: starting
0501 13:58:38.352365 (+ 60us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:38.352467 (+ 102us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:38.352564 (+ 97us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:38.353007 (+ 443us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:38.353148 (+ 141us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:38.353193 (+ 45us) write_op.cc:312] APPLY: finished
0501 13:58:38.353386 (+ 193us) log.cc:844] Serialized 29 byte log entry
0501 13:58:38.353539 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:38.353611 (+ 72us) write_op.cc:454] Released schema lock
0501 13:58:38.353641 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":84,"Hive Metastore.run_cpu_time_us":619,"Hive Metastore.run_wall_time_us":280009,"child_traces":[["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":1534,"apply.run_wall_time_us":1534,"num_ops":2,"prepare.queue_time_us":3809,"prepare.run_cpu_time_us":1939,"prepare.run_wall_time_us":3936,"raft.queue_time_us":966,"raft.run_cpu_time_us":16,"raft.run_wall_time_us":16,"replication_time_us":23905}]]}
I20260501 13:58:38.365073 4055 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_aafd3ab3f9f44a74af41aae60cfe40fa" } modify_external_catalogs: true
I20260501 13:58:38.366223 4055 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_aafd3ab3f9f44a74af41aae60cfe40fa" } modify_external_catalogs: true
I20260501 13:58:38.375165 4357 ts_tablet_manager.cc:1916] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:38.384752 4359 tablet_replica.cc:333] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:38.384894 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]) successfully deleted
I20260501 13:58:38.385458 4359 raft_consensus.cc:2243] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:38.386106 4359 pending_rounds.cc:70] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: Trying to abort 1 pending ops.
I20260501 13:58:38.386477 4359 pending_rounds.cc:77] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229487392440320 op_type: NO_OP noop_request { }
I20260501 13:58:38.386826 4359 raft_consensus.cc:2889] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:38.387137 4359 raft_consensus.cc:2272] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:58:38.387285 3757 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.129:44335, user_credentials={real_user=slave}} blocked reactor thread for 93554.2us
I20260501 13:58:38.397552 4359 ts_tablet_manager.cc:1916] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:38.397533 3493 tablet_service.cc:1511] Processing CreateTablet for tablet 533eb8337c414aea98ea95d7bf507b1e (DEFAULT_TABLE table=default.table_f52f85ad11fb4023adead2ab9c5eb694 [id=53fa808a27054cdf834732393dc08838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.403481 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533eb8337c414aea98ea95d7bf507b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.411497 4056 master_service.cc:954] ReplaceTablet: received request to replace tablet 145974ff0af1418e839bf81b88d9f205 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:38.389148 4338 tablet_bootstrap.cc:654] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:38.416890 4066 catalog_manager.cc:5671] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:38.422554 3491 tablet_service.cc:1511] Processing CreateTablet for tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 (DEFAULT_TABLE table=default.table_0da08f161093429e83b826825c3a9727 [id=5b3f55db8abb4fa7a9b3abf286620942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.423704 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 151c6fb3afbd46a9a4c9cac128ad4ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.424997 4370 raft_consensus.cc:493] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:38.425475 4370 raft_consensus.cc:515] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.427260 4359 ts_tablet_manager.cc:1929] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:38.427611 4359 log.cc:1199] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/145974ff0af1418e839bf81b88d9f205
I20260501 13:58:38.428273 4359 ts_tablet_manager.cc:1950] T 145974ff0af1418e839bf81b88d9f205 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:38.429759 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6234730e704c77a3a85a407f0e1cff" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:38.430223 3654 tablet_service.cc:1511] Processing CreateTablet for tablet 533eb8337c414aea98ea95d7bf507b1e (DEFAULT_TABLE table=default.table_f52f85ad11fb4023adead2ab9c5eb694 [id=53fa808a27054cdf834732393dc08838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.431259 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533eb8337c414aea98ea95d7bf507b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.431391 3800 tablet_service.cc:1511] Processing CreateTablet for tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 (DEFAULT_TABLE table=default.table_0da08f161093429e83b826825c3a9727 [id=5b3f55db8abb4fa7a9b3abf286620942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.432488 3800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 151c6fb3afbd46a9a4c9cac128ad4ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.435191 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 533eb8337c414aea98ea95d7bf507b1e (DEFAULT_TABLE table=default.table_f52f85ad11fb4023adead2ab9c5eb694 [id=53fa808a27054cdf834732393dc08838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.436160 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533eb8337c414aea98ea95d7bf507b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.438769 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet 02102952baaf40e0b897bc6dc04426da with delete_type TABLET_DATA_DELETED (replaced by tablet 41ac3120d65440829fe929b76cc41d43) from {username='slave'} at 127.0.0.1:47972
W20260501 13:58:38.440244 3448 leader_election.cc:343] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:38.440745 3651 tablet_service.cc:1511] Processing CreateTablet for tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 (DEFAULT_TABLE table=default.table_0da08f161093429e83b826825c3a9727 [id=5b3f55db8abb4fa7a9b3abf286620942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.441934 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 151c6fb3afbd46a9a4c9cac128ad4ec0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:58:38.452633 4056 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:38.455363 4370 leader_election.cc:290] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:38.456943 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6234730e704c77a3a85a407f0e1cff" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:38.457532 3822 raft_consensus.cc:2468] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:38.459146 3448 leader_election.cc:304] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:38.460336 4370 raft_consensus.cc:2804] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:38.460698 4370 raft_consensus.cc:493] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:38.461033 4370 raft_consensus.cc:3060] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:38.462436 4338 tablet_bootstrap.cc:492] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:38.462916 4338 ts_tablet_manager.cc:1403] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.313s user 0.010s sys 0.000s
I20260501 13:58:38.465440 4338 raft_consensus.cc:359] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.466096 4338 raft_consensus.cc:385] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.466347 4338 raft_consensus.cc:740] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:38.462996 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 145974ff0af1418e839bf81b88d9f205 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
W20260501 13:58:38.469516 4036 catalog_manager.cc:4969] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet 02102952baaf40e0b897bc6dc04426da because the tablet was not found. No further retry: Not found: Tablet not found: 02102952baaf40e0b897bc6dc04426da
I20260501 13:58:38.476325 4338 consensus_queue.cc:260] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.480602 4338 ts_tablet_manager.cc:1434] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260501 13:58:38.481537 4338 tablet_bootstrap.cc:492] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:38.482635 4341 tablet_bootstrap.cc:492] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:38.487426 4421 raft_consensus.cc:493] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:38.487920 4421 raft_consensus.cc:515] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.489697 4421 leader_election.cc:290] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:38.491768 4341 tablet_bootstrap.cc:654] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:38.491551 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6234730e704c77a3a85a407f0e1cff" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:38.493089 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6234730e704c77a3a85a407f0e1cff" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:38.493634 3822 raft_consensus.cc:2468] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:38.494882 3608 leader_election.cc:304] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:38.494905 4370 raft_consensus.cc:515] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.495805 4421 raft_consensus.cc:2804] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:38.496007 3513 raft_consensus.cc:2393] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in current term 1: Already voted for candidate 9154033cea374a10a5225c53b8bcf122 in this term.
I20260501 13:58:38.496133 4421 raft_consensus.cc:493] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:38.496414 4421 raft_consensus.cc:3060] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:38.506614 4433 tablet_bootstrap.cc:492] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:38.512840 4433 tablet_bootstrap.cc:654] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:38.516255 4421 raft_consensus.cc:515] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.520819 4341 tablet_bootstrap.cc:492] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:38.521238 4341 ts_tablet_manager.cc:1403] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.000s
I20260501 13:58:38.523519 4341 raft_consensus.cc:359] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.524117 4341 raft_consensus.cc:385] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.524360 4341 raft_consensus.cc:740] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:38.527475 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6234730e704c77a3a85a407f0e1cff" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:38.527576 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6234730e704c77a3a85a407f0e1cff" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:38.527916 4338 tablet_bootstrap.cc:654] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:38.528069 3822 raft_consensus.cc:3060] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:38.528205 3513 raft_consensus.cc:2393] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in current term 1: Already voted for candidate 9154033cea374a10a5225c53b8bcf122 in this term.
I20260501 13:58:38.529542 4341 consensus_queue.cc:260] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.536669 3822 raft_consensus.cc:2468] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:38.538276 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6234730e704c77a3a85a407f0e1cff" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:38.539057 3672 raft_consensus.cc:2393] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9154033cea374a10a5225c53b8bcf122 in current term 1: Already voted for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in this term.
I20260501 13:58:38.538849 3608 leader_election.cc:304] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:38.540604 4341 ts_tablet_manager.cc:1434] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260501 13:58:38.540771 4382 raft_consensus.cc:2804] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:38.541199 4382 raft_consensus.cc:697] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:38.542029 4382 consensus_queue.cc:237] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.544189 4370 leader_election.cc:290] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:38.544793 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6234730e704c77a3a85a407f0e1cff" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:38.545485 3822 raft_consensus.cc:2393] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9154033cea374a10a5225c53b8bcf122 in current term 1: Already voted for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in this term.
I20260501 13:58:38.549224 4341 tablet_bootstrap.cc:492] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:38.556322 3448 leader_election.cc:304] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:38.557595 4370 raft_consensus.cc:2749] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260501 13:58:38.562952 4066 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: default.table_04a1387d0f0343998d94c1bb5bbff699 [id=18ac75baaaf7485da21228e06a780ad2] alter complete (version 1)
I20260501 13:58:38.563740 4421 leader_election.cc:290] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:38.566114 4056 master_service.cc:954] ReplaceTablet: received request to replace tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:38.592592 4054 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_82ebd50001e848a8b155dead3f78ea41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:38.587304 4357 ts_tablet_manager.cc:1929] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:38.598912 4357 log.cc:1199] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/145974ff0af1418e839bf81b88d9f205
I20260501 13:58:38.599571 4357 ts_tablet_manager.cc:1950] T 145974ff0af1418e839bf81b88d9f205 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:38.602087 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 145974ff0af1418e839bf81b88d9f205 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
I20260501 13:58:38.623687 3800 tablet_service.cc:1558] Processing DeleteTablet for tablet 145974ff0af1418e839bf81b88d9f205 with delete_type TABLET_DATA_DELETED (replaced by tablet d6c11a5794f64e69a3f467800a7cd2ae) from {username='slave'} at 127.0.0.1:47972
W20260501 13:58:38.626678 4036 catalog_manager.cc:5075] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): alter failed for tablet 145974ff0af1418e839bf81b88d9f205 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]),no further retry: Not found: Tablet not found: 145974ff0af1418e839bf81b88d9f205
W20260501 13:58:38.627405 4036 catalog_manager.cc:4969] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): delete failed for tablet 145974ff0af1418e839bf81b88d9f205 because the tablet was not found. No further retry: Not found: Tablet not found: 145974ff0af1418e839bf81b88d9f205
I20260501 13:58:38.629271 4056 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3
I20260501 13:58:38.630744 4056 catalog_manager.cc:6741] ReplaceTablet: tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3 deleted and replaced by tablet fb0b2d069e75401ba56839e072156ac7
I20260501 13:58:38.632544 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3 with delete_type TABLET_DATA_DELETED (replaced by tablet fb0b2d069e75401ba56839e072156ac7) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:38.633371 4357 tablet_replica.cc:333] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:38.634142 4357 raft_consensus.cc:2243] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:38.634666 4357 raft_consensus.cc:2272] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:38.636475 4357 ts_tablet_manager.cc:1916] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:38.644735 4433 tablet_bootstrap.cc:492] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:38.645229 4433 ts_tablet_manager.cc:1403] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.139s user 0.012s sys 0.001s
I20260501 13:58:38.653975 4338 tablet_bootstrap.cc:492] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:38.655606 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3 with delete_type TABLET_DATA_DELETED (replaced by tablet fb0b2d069e75401ba56839e072156ac7) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:38.655838 3800 tablet_service.cc:1558] Processing DeleteTablet for tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3 with delete_type TABLET_DATA_DELETED (replaced by tablet fb0b2d069e75401ba56839e072156ac7) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:38.656354 4358 tablet_replica.cc:333] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:38.657033 4358 raft_consensus.cc:2243] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:38.657383 4359 tablet_replica.cc:333] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:38.657460 4358 raft_consensus.cc:2272] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:38.658102 4359 raft_consensus.cc:2243] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:38.658783 4359 pending_rounds.cc:70] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: Trying to abort 1 pending ops.
I20260501 13:58:38.658993 4359 pending_rounds.cc:77] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229489210482688 op_type: NO_OP noop_request { }
I20260501 13:58:38.659159 4358 ts_tablet_manager.cc:1916] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:38.659305 4359 raft_consensus.cc:2889] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:38.659401 4338 ts_tablet_manager.cc:1403] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.173s user 0.010s sys 0.000s
I20260501 13:58:38.659610 4359 raft_consensus.cc:2272] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:38.661835 4359 ts_tablet_manager.cc:1916] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:38.663455 4341 tablet_bootstrap.cc:654] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:38.671433 4066 master_service.cc:954] ReplaceTablet: received request to replace tablet 80a2f6fab87a4ad980c7816f34f41aff from {username='slave'} at 127.0.0.1:35438
W20260501 13:58:35.178828 4351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 13:58:36.576226 4348 debug-util.cc:398] Leaking SignalData structure 0x7b080002ecc0 after lost signal to thread 4309
W20260501 13:58:38.678030 4348 kernel_stack_watchdog.cc:198] Thread 4309 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20260501 13:58:38.679802 3493 tablet_service.cc:1511] Processing CreateTablet for tablet addbcaae1b464378aabe7bc7f4bcc7fa (DEFAULT_TABLE table=default.table_c0462a887c864245890f76dea029e11b [id=52fc296b289947f1be9e9f74d320675c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.679776 4056 catalog_manager.cc:5671] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:38.680840 3493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addbcaae1b464378aabe7bc7f4bcc7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.682040 4359 ts_tablet_manager.cc:1929] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:38.682385 4359 log.cc:1199] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/dfea0ce6458e4d5a8e9a5dbe87fa28c3
I20260501 13:58:38.683929 3491 tablet_service.cc:1511] Processing CreateTablet for tablet 41ac3120d65440829fe929b76cc41d43 (DEFAULT_TABLE table=default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.684590 3800 tablet_service.cc:1511] Processing CreateTablet for tablet addbcaae1b464378aabe7bc7f4bcc7fa (DEFAULT_TABLE table=default.table_c0462a887c864245890f76dea029e11b [id=52fc296b289947f1be9e9f74d320675c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.685017 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ac3120d65440829fe929b76cc41d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.652243 4433 raft_consensus.cc:359] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.685637 3800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addbcaae1b464378aabe7bc7f4bcc7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.685705 4433 raft_consensus.cc:385] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.685978 4433 raft_consensus.cc:740] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:38.686669 4433 consensus_queue.cc:260] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.687263 3654 tablet_service.cc:1511] Processing CreateTablet for tablet addbcaae1b464378aabe7bc7f4bcc7fa (DEFAULT_TABLE table=default.table_c0462a887c864245890f76dea029e11b [id=52fc296b289947f1be9e9f74d320675c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.688338 3654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addbcaae1b464378aabe7bc7f4bcc7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.688894 4433 ts_tablet_manager.cc:1434] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20260501 13:58:38.713037 3492 tablet_service.cc:1511] Processing CreateTablet for tablet d6c11a5794f64e69a3f467800a7cd2ae (DEFAULT_TABLE table=default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.713145 4338 raft_consensus.cc:359] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.713805 4338 raft_consensus.cc:385] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.714053 4338 raft_consensus.cc:740] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:38.714198 3492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c11a5794f64e69a3f467800a7cd2ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:58:38.714969 4037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260501 13:58:38.716560 3651 tablet_service.cc:1511] Processing CreateTablet for tablet 41ac3120d65440829fe929b76cc41d43 (DEFAULT_TABLE table=default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.717653 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ac3120d65440829fe929b76cc41d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.720621 4341 tablet_bootstrap.cc:492] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:38.721081 4341 ts_tablet_manager.cc:1403] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.172s user 0.008s sys 0.000s
I20260501 13:58:38.723459 4341 raft_consensus.cc:359] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.724074 4359 ts_tablet_manager.cc:1950] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:38.724074 4341 raft_consensus.cc:385] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.724452 4341 raft_consensus.cc:740] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:38.728441 4059 catalog_manager.cc:2174] created HMS catalog entry for table [id=4f022a0cee6540b5959fd48b029df1af]
I20260501 13:58:38.728082 3800 tablet_service.cc:1511] Processing CreateTablet for tablet 41ac3120d65440829fe929b76cc41d43 (DEFAULT_TABLE table=default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.733796 3800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ac3120d65440829fe929b76cc41d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.733541 4341 consensus_queue.cc:260] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:38.736930 4309 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 3.565s user 0.183s sys 0.218s
W20260501 13:58:38.737869 4309 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 3.566s user 0.184s sys 0.218s
I20260501 13:58:38.747694 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3 (table default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]) successfully deleted
I20260501 13:58:38.751279 4059 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 55) took 2323 ms. Trace:
I20260501 13:58:38.751658 4059 rpcz_store.cc:276] 0501 13:58:36.427678 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:36.434436 (+ 6758us) service_pool.cc:224] Handling call
0501 13:58:38.049603 (+1615167us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:38.051838 (+ 2235us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:38.728393 (+676555us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:38.750612 (+ 22219us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:38.750940 (+ 328us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:38.751214 (+ 274us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:38.730381 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:38.730812 (+ 431us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:38.730836 (+ 24us) write_op.cc:435] Acquired schema lock
0501 13:58:38.730848 (+ 12us) tablet.cc:662] Decoding operations
0501 13:58:38.731097 (+ 249us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:38.731137 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:38.731160 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:38.731338 (+ 178us) tablet.cc:701] Row locks acquired
0501 13:58:38.731351 (+ 13us) write_op.cc:260] PREPARE: finished
0501 13:58:38.731472 (+ 121us) write_op.cc:270] Start()
0501 13:58:38.731563 (+ 91us) write_op.cc:276] Timestamp: P: 1777643918731422 usec, L: 0
0501 13:58:38.731579 (+ 16us) op_driver.cc:348] REPLICATION: starting
0501 13:58:38.732058 (+ 479us) log.cc:844] Serialized 433 byte log entry
0501 13:58:38.747969 (+ 15911us) op_driver.cc:464] REPLICATION: finished
0501 13:58:38.748358 (+ 389us) write_op.cc:301] APPLY: starting
0501 13:58:38.748416 (+ 58us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:38.748527 (+ 111us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:38.748611 (+ 84us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:38.748972 (+ 361us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:38.749108 (+ 136us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:38.749154 (+ 46us) write_op.cc:312] APPLY: finished
0501 13:58:38.749337 (+ 183us) log.cc:844] Serialized 29 byte log entry
0501 13:58:38.749477 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:38.749554 (+ 77us) write_op.cc:454] Released schema lock
0501 13:58:38.749587 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:38.749799 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:38.749821 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":268227,"Hive Metastore.run_cpu_time_us":621,"Hive Metastore.run_wall_time_us":407461,"child_traces":[["op",{"apply.queue_time_us":241,"apply.run_cpu_time_us":1413,"apply.run_wall_time_us":1412,"num_ops":2,"prepare.queue_time_us":128,"prepare.run_cpu_time_us":2004,"prepare.run_wall_time_us":2004,"raft.queue_time_us":466,"raft.run_cpu_time_us":662,"raft.run_wall_time_us":662,"replication_time_us":16301}]]}
I20260501 13:58:38.755198 4338 consensus_queue.cc:260] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.760232 4309 server_base.cc:1061] running on GCE node
I20260501 13:58:38.764869 4341 ts_tablet_manager.cc:1434] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.044s user 0.007s sys 0.001s
I20260501 13:58:38.771814 4433 tablet_bootstrap.cc:492] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:38.779781 3652 tablet_service.cc:1511] Processing CreateTablet for tablet d6c11a5794f64e69a3f467800a7cd2ae (DEFAULT_TABLE table=default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.780789 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c11a5794f64e69a3f467800a7cd2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.783247 3802 tablet_service.cc:1511] Processing CreateTablet for tablet d6c11a5794f64e69a3f467800a7cd2ae (DEFAULT_TABLE table=default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.784227 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c11a5794f64e69a3f467800a7cd2ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:58:38.786258 4474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 13:58:38.788411 4338 ts_tablet_manager.cc:1434] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.129s user 0.005s sys 0.000s
I20260501 13:58:38.792155 4309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 13:58:38.800595 4341 tablet_bootstrap.cc:492] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:38.803197 4338 tablet_bootstrap.cc:492] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
W20260501 13:58:38.807709 4052 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8be6ed54e88344f7b4be783af8fa48da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:58:38.809613 4064 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_59dc65b11ade4ee3acb91116f7a60585 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:38.811219 4358 ts_tablet_manager.cc:1929] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:38.811995 4358 log.cc:1199] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/dfea0ce6458e4d5a8e9a5dbe87fa28c3
I20260501 13:58:38.813031 4050 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35438 (request call id 43) took 4101 ms. Trace:
I20260501 13:58:38.813371 4050 rpcz_store.cc:276] 0501 13:58:34.711242 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:34.711381 (+ 139us) service_pool.cc:224] Handling call
0501 13:58:36.558340 (+1846959us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:38.808761 (+2250421us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:38.812151 (+ 3390us) catalog_manager.cc:3725] Apply alter schema
0501 13:58:38.812964 (+ 813us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":2320,"Hive Metastore.run_cpu_time_us":1827,"Hive Metastore.run_wall_time_us":426657,"mutex_wait_us":30008}
I20260501 13:58:38.817946 4338 tablet_bootstrap.cc:654] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:38.825174 4309 hybrid_clock.cc:648] HybridClock initialized: now 1777643918825074 us; error 111 us; skew 500 ppm
I20260501 13:58:38.826329 4309 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260501 13:58:38.829505 4341 tablet_bootstrap.cc:654] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:38.813252 4358 ts_tablet_manager.cc:1950] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:38.856679 4338 tablet_bootstrap.cc:492] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:38.857125 4338 ts_tablet_manager.cc:1403] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.004s
I20260501 13:58:38.860004 4050 master_service.cc:954] ReplaceTablet: received request to replace tablet 80a2f6fab87a4ad980c7816f34f41aff from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:38.869807 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3 (table default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]) successfully deleted
I20260501 13:58:38.872650 4341 tablet_bootstrap.cc:492] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:38.873052 4341 ts_tablet_manager.cc:1403] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.016s
I20260501 13:58:38.875370 4341 raft_consensus.cc:359] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.876014 4341 raft_consensus.cc:385] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.876261 4341 raft_consensus.cc:740] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:38.876950 4341 consensus_queue.cc:260] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:38.883549 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 6e76ac9fdeed442b94510860cd439b24 (DEFAULT_TABLE table=default.table_124cdf87baa148fbbc6e5f06d1efcda1 [id=4f022a0cee6540b5959fd48b029df1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.884596 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e76ac9fdeed442b94510860cd439b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.885739 4338 raft_consensus.cc:359] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.888294 4050 master_service.cc:954] ReplaceTablet: received request to replace tablet ba69864093504babbb00fb6170774778 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:38.889115 3492 tablet_service.cc:1511] Processing CreateTablet for tablet 6e76ac9fdeed442b94510860cd439b24 (DEFAULT_TABLE table=default.table_124cdf87baa148fbbc6e5f06d1efcda1 [id=4f022a0cee6540b5959fd48b029df1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.889810 3651 tablet_service.cc:1511] Processing CreateTablet for tablet fb0b2d069e75401ba56839e072156ac7 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.890296 3492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e76ac9fdeed442b94510860cd439b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.890975 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb0b2d069e75401ba56839e072156ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.892375 4309 webserver.cc:492] Webserver started at http://127.1.16.189:36257/ using document root <none> and password file <none>
I20260501 13:58:38.893674 4309 fs_manager.cc:362] Metadata directory not provided
I20260501 13:58:38.893986 4309 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 13:58:38.895139 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 6e76ac9fdeed442b94510860cd439b24 (DEFAULT_TABLE table=default.table_124cdf87baa148fbbc6e5f06d1efcda1 [id=4f022a0cee6540b5959fd48b029df1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.896164 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e76ac9fdeed442b94510860cd439b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.900022 4341 ts_tablet_manager.cc:1434] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260501 13:58:38.901554 3491 tablet_service.cc:1511] Processing CreateTablet for tablet fb0b2d069e75401ba56839e072156ac7 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.901527 3800 tablet_service.cc:1511] Processing CreateTablet for tablet fb0b2d069e75401ba56839e072156ac7 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:38.902679 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb0b2d069e75401ba56839e072156ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.902772 3800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb0b2d069e75401ba56839e072156ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:38.908804 4433 tablet_bootstrap.cc:654] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:38.886360 4338 raft_consensus.cc:385] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.915480 4338 raft_consensus.cc:740] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:38.916162 4338 consensus_queue.cc:260] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.924026 4341 tablet_bootstrap.cc:492] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:38.929523 4341 tablet_bootstrap.cc:654] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:38.933209 4357 ts_tablet_manager.cc:1929] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:38.936400 4309 fs_manager.cc:714] Time spent opening directory manager: real 0.034s user 0.005s sys 0.001s
I20260501 13:58:38.937083 4338 ts_tablet_manager.cc:1434] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.080s user 0.008s sys 0.000s
I20260501 13:58:38.938184 4338 tablet_bootstrap.cc:492] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:38.946524 4357 log.cc:1199] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/dfea0ce6458e4d5a8e9a5dbe87fa28c3
I20260501 13:58:38.951026 4357 ts_tablet_manager.cc:1950] T dfea0ce6458e4d5a8e9a5dbe87fa28c3 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:38.955680 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet dfea0ce6458e4d5a8e9a5dbe87fa28c3 (table default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]) successfully deleted
I20260501 13:58:38.961237 4481 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 13:58:38.964607 4059 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_9074c6e0ff374999bcf7953f17de8b4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:38.972992 4338 tablet_bootstrap.cc:654] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:38.974347 4309 fs_manager.cc:730] Time spent opening block manager: real 0.034s user 0.006s sys 0.000s
I20260501 13:58:38.974867 4309 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data,/tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal
uuid: "15dd70d7c5494c3c911f0a780dc9a2af"
format_stamp: "Formatted at 2026-05-01 13:58:11 on dist-test-slave-nqxd"
I20260501 13:58:38.977834 4309 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 13:58:38.980342 4433 tablet_bootstrap.cc:492] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:38.980818 4433 ts_tablet_manager.cc:1403] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.209s user 0.009s sys 0.001s
I20260501 13:58:38.995126 4341 tablet_bootstrap.cc:492] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:38.995628 4341 ts_tablet_manager.cc:1403] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.001s
I20260501 13:58:38.998118 4341 raft_consensus.cc:359] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.987656 4433 raft_consensus.cc:359] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:38.999120 4433 raft_consensus.cc:385] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:38.999403 4433 raft_consensus.cc:740] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.000140 4433 consensus_queue.cc:260] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.000910 4341 raft_consensus.cc:385] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.001216 4341 raft_consensus.cc:740] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.007006 4341 consensus_queue.cc:260] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.015914 4341 ts_tablet_manager.cc:1434] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260501 13:58:39.016973 4066 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:39.019348 4341 tablet_bootstrap.cc:492] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
W20260501 13:58:39.018456 4066 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 41ac3120d65440829fe929b76cc41d43 consensus state has no leader
I20260501 13:58:39.022397 4056 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_f701ba5ffc3845bf8cb5e55102e60688" } new_table_name: "default.Table_3240c40926184fd7b92c22b821de47e9" modify_external_catalogs: true
W20260501 13:58:39.035322 4050 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:39.035617 4050 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
I20260501 13:58:39.038326 4066 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:39.042196 4341 tablet_bootstrap.cc:654] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.045958 4370 raft_consensus.cc:493] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:39.046502 4370 raft_consensus.cc:515] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.048434 4370 leader_election.cc:290] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:39.052771 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "533eb8337c414aea98ea95d7bf507b1e" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:39.053426 3822 raft_consensus.cc:2468] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
W20260501 13:58:39.055231 4038 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 41ac3120d65440829fe929b76cc41d43 consensus state has no leader
I20260501 13:58:39.059473 3448 leader_election.cc:304] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:39.060249 4370 raft_consensus.cc:2804] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:39.061913 4433 ts_tablet_manager.cc:1434] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.081s user 0.007s sys 0.001s
I20260501 13:58:39.065977 4341 tablet_bootstrap.cc:492] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:39.066515 4341 ts_tablet_manager.cc:1403] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.002s
I20260501 13:58:39.060596 4370 raft_consensus.cc:493] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:39.069175 4370 raft_consensus.cc:3060] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.068979 4341 raft_consensus.cc:359] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.071149 4341 raft_consensus.cc:385] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.071462 4341 raft_consensus.cc:740] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.074987 4433 tablet_bootstrap.cc:492] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:39.075809 4341 consensus_queue.cc:260] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.084671 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "533eb8337c414aea98ea95d7bf507b1e" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:39.086091 3672 raft_consensus.cc:2468] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:39.088745 4433 tablet_bootstrap.cc:654] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.089815 4063 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]
I20260501 13:58:39.095985 4066 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet d6c11a5794f64e69a3f467800a7cd2ae (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]). Expected version 1 got 0
I20260501 13:58:39.094302 4421 raft_consensus.cc:493] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:39.099756 4421 raft_consensus.cc:515] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.103631 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "151c6fb3afbd46a9a4c9cac128ad4ec0" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:39.104035 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "151c6fb3afbd46a9a4c9cac128ad4ec0" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:39.104281 3513 raft_consensus.cc:2468] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:39.102380 4421 leader_election.cc:290] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:39.104615 3822 raft_consensus.cc:2468] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:39.105881 3608 leader_election.cc:304] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:39.107645 4341 ts_tablet_manager.cc:1434] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.041s user 0.003s sys 0.002s
I20260501 13:58:39.108273 4338 tablet_bootstrap.cc:492] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:39.108965 4338 ts_tablet_manager.cc:1403] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.171s user 0.016s sys 0.017s
W20260501 13:58:39.110745 4066 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet d6c11a5794f64e69a3f467800a7cd2ae (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d6c11a5794f64e69a3f467800a7cd2ae consensus state has no leader
I20260501 13:58:39.118055 4050 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet d6c11a5794f64e69a3f467800a7cd2ae (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]). Expected version 1 got 0
I20260501 13:58:39.121073 4338 raft_consensus.cc:359] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.121834 4338 raft_consensus.cc:385] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.122078 4341 tablet_bootstrap.cc:492] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:39.122129 4338 raft_consensus.cc:740] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.127550 4338 consensus_queue.cc:260] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.133352 4338 ts_tablet_manager.cc:1434] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.024s user 0.004s sys 0.005s
I20260501 13:58:39.134629 4338 tablet_bootstrap.cc:492] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:39.145102 4370 raft_consensus.cc:515] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.146987 4050 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:39.149948 4341 tablet_bootstrap.cc:654] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.170961 4050 master_service.cc:954] ReplaceTablet: received request to replace tablet a9ecb5b51ae04f32a4f07f828d563fa4 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:39.174875 4066 master_service.cc:954] ReplaceTablet: received request to replace tablet d5862fb4c6ca43feb61939b8698e728c from {username='slave'} at 127.0.0.1:35438
W20260501 13:58:39.179677 4037 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 -> Peer 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255): Couldn't send request to peer 15dd70d7c5494c3c911f0a780dc9a2af. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260501 13:58:39.195576 4066 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet d5862fb4c6ca43feb61939b8698e728c
I20260501 13:58:39.196914 4066 catalog_manager.cc:6741] ReplaceTablet: tablet d5862fb4c6ca43feb61939b8698e728c deleted and replaced by tablet 776eacf49dbd4c2f9a8dfa1ae2181c96
I20260501 13:58:39.198613 3491 tablet_service.cc:1558] Processing DeleteTablet for tablet d5862fb4c6ca43feb61939b8698e728c with delete_type TABLET_DATA_DELETED (replaced by tablet 776eacf49dbd4c2f9a8dfa1ae2181c96) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:39.199352 4357 tablet_replica.cc:333] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:39.200127 4357 raft_consensus.cc:2243] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:39.200380 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet d5862fb4c6ca43feb61939b8698e728c with delete_type TABLET_DATA_DELETED (replaced by tablet 776eacf49dbd4c2f9a8dfa1ae2181c96) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:39.200631 4357 raft_consensus.cc:2272] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:39.201102 4359 tablet_replica.cc:333] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:39.201781 4359 raft_consensus.cc:2243] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:39.202631 4359 raft_consensus.cc:2272] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:39.204396 4357 ts_tablet_manager.cc:1916] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:39.206646 4370 leader_election.cc:290] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:39.206503 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "533eb8337c414aea98ea95d7bf507b1e" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:39.207109 3822 raft_consensus.cc:3060] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.211966 4341 tablet_bootstrap.cc:492] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:39.206449 4359 ts_tablet_manager.cc:1916] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:39.212440 4341 ts_tablet_manager.cc:1403] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.091s user 0.008s sys 0.004s
I20260501 13:58:39.219112 3822 raft_consensus.cc:2468] T 533eb8337c414aea98ea95d7bf507b1e P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:39.220415 3448 leader_election.cc:304] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
W20260501 13:58:39.221617 4066 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:39.223698 4341 raft_consensus.cc:359] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.226879 4341 raft_consensus.cc:385] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.227205 4341 raft_consensus.cc:740] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.227892 4341 consensus_queue.cc:260] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.231211 4370 raft_consensus.cc:2804] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
W20260501 13:58:39.235916 4037 catalog_manager.cc:5075] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): alter failed for tablet 02102952baaf40e0b897bc6dc04426da (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]),no further retry: Not found: Tablet not found: 02102952baaf40e0b897bc6dc04426da
I20260501 13:58:39.241770 4341 ts_tablet_manager.cc:1434] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.029s user 0.007s sys 0.001s
I20260501 13:58:39.247992 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "533eb8337c414aea98ea95d7bf507b1e" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:39.248608 3672 raft_consensus.cc:3060] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.251610 3652 tablet_service.cc:1558] Processing DeleteTablet for tablet d5862fb4c6ca43feb61939b8698e728c with delete_type TABLET_DATA_DELETED (replaced by tablet 776eacf49dbd4c2f9a8dfa1ae2181c96) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:39.252586 4358 tablet_replica.cc:333] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:39.253249 4358 raft_consensus.cc:2243] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:39.256100 4358 raft_consensus.cc:2272] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:39.259460 4341 tablet_bootstrap.cc:492] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:39.261044 4338 tablet_bootstrap.cc:654] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.261375 4358 ts_tablet_manager.cc:1916] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:39.268235 4357 ts_tablet_manager.cc:1929] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:39.268622 4357 log.cc:1199] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/d5862fb4c6ca43feb61939b8698e728c
I20260501 13:58:39.269272 4357 ts_tablet_manager.cc:1950] T d5862fb4c6ca43feb61939b8698e728c P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:39.274101 3672 raft_consensus.cc:2468] T 533eb8337c414aea98ea95d7bf507b1e P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:39.111491 4487 raft_consensus.cc:2804] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:39.275749 4487 raft_consensus.cc:493] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:39.276099 4487 raft_consensus.cc:3060] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.277966 4341 tablet_bootstrap.cc:654] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.279150 4359 ts_tablet_manager.cc:1929] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:39.283281 4338 tablet_bootstrap.cc:492] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:39.231680 4370 raft_consensus.cc:697] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:39.283427 4358 ts_tablet_manager.cc:1929] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:39.283830 4358 log.cc:1199] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/d5862fb4c6ca43feb61939b8698e728c
I20260501 13:58:39.283937 4338 ts_tablet_manager.cc:1403] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.150s user 0.009s sys 0.003s
I20260501 13:58:39.284669 4359 log.cc:1199] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/d5862fb4c6ca43feb61939b8698e728c
I20260501 13:58:39.286362 4433 tablet_bootstrap.cc:492] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:39.284205 4370 consensus_queue.cc:237] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.287182 4433 ts_tablet_manager.cc:1403] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.212s user 0.036s sys 0.016s
I20260501 13:58:39.287122 4338 raft_consensus.cc:359] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.287914 4338 raft_consensus.cc:385] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.288210 4338 raft_consensus.cc:740] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.288241 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet d5862fb4c6ca43feb61939b8698e728c (table default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]) successfully deleted
I20260501 13:58:39.288944 4338 consensus_queue.cc:260] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.290222 4433 raft_consensus.cc:359] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.290954 4433 raft_consensus.cc:385] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.291224 4433 raft_consensus.cc:740] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.292850 4338 ts_tablet_manager.cc:1434] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260501 13:58:39.300757 4433 consensus_queue.cc:260] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.301298 4359 ts_tablet_manager.cc:1950] T d5862fb4c6ca43feb61939b8698e728c P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:39.303035 4433 ts_tablet_manager.cc:1434] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260501 13:58:39.304054 4433 tablet_bootstrap.cc:492] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:39.310242 4341 tablet_bootstrap.cc:492] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:39.311023 4341 ts_tablet_manager.cc:1403] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.004s
I20260501 13:58:39.316666 4433 tablet_bootstrap.cc:654] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.317510 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet d5862fb4c6ca43feb61939b8698e728c (table default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]) successfully deleted
I20260501 13:58:39.320814 4443 raft_consensus.cc:493] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:39.321275 4443 raft_consensus.cc:515] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.323129 4443 leader_election.cc:290] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:39.293927 4358 ts_tablet_manager.cc:1950] T d5862fb4c6ca43feb61939b8698e728c P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:39.324462 4338 tablet_bootstrap.cc:492] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:39.327574 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet d5862fb4c6ca43feb61939b8698e728c (table default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]) successfully deleted
I20260501 13:58:39.327948 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb0b2d069e75401ba56839e072156ac7" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:39.328274 4341 raft_consensus.cc:359] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.328882 4341 raft_consensus.cc:385] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.329135 4341 raft_consensus.cc:740] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
W20260501 13:58:39.329825 3757 leader_election.cc:343] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:39.329686 4341 consensus_queue.cc:260] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.331583 4341 ts_tablet_manager.cc:1434] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260501 13:58:39.334646 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb0b2d069e75401ba56839e072156ac7" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
W20260501 13:58:39.353873 3756 leader_election.cc:343] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:39.354286 3756 leader_election.cc:304] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:39.355240 4443 raft_consensus.cc:2749] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:39.355365 4487 raft_consensus.cc:515] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.361763 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "151c6fb3afbd46a9a4c9cac128ad4ec0" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:39.362285 3822 raft_consensus.cc:3060] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.364415 4433 tablet_bootstrap.cc:492] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:39.364845 4433 ts_tablet_manager.cc:1403] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.000s
I20260501 13:58:39.368057 4433 raft_consensus.cc:359] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.369524 3822 raft_consensus.cc:2468] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:39.370826 3608 leader_election.cc:304] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:39.373178 4433 raft_consensus.cc:385] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.373562 4433 raft_consensus.cc:740] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.375452 4382 raft_consensus.cc:2804] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:39.376029 4382 raft_consensus.cc:697] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:39.375082 4433 consensus_queue.cc:260] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.376890 4382 consensus_queue.cc:237] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.379202 4433 ts_tablet_manager.cc:1434] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260501 13:58:39.380185 4433 tablet_bootstrap.cc:492] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:39.381798 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "151c6fb3afbd46a9a4c9cac128ad4ec0" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:39.382332 3513 raft_consensus.cc:3060] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.389742 3513 raft_consensus.cc:2468] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:39.394621 4487 leader_election.cc:290] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:39.396076 4433 tablet_bootstrap.cc:654] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.408784 3491 tablet_service.cc:1511] Processing CreateTablet for tablet 776eacf49dbd4c2f9a8dfa1ae2181c96 (DEFAULT_TABLE table=default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:39.409575 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 776eacf49dbd4c2f9a8dfa1ae2181c96 (DEFAULT_TABLE table=default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:39.409977 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776eacf49dbd4c2f9a8dfa1ae2181c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:39.410887 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776eacf49dbd4c2f9a8dfa1ae2181c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:39.413409 4338 tablet_bootstrap.cc:654] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.417423 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 776eacf49dbd4c2f9a8dfa1ae2181c96 (DEFAULT_TABLE table=default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:39.418486 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776eacf49dbd4c2f9a8dfa1ae2181c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:39.419122 4443 raft_consensus.cc:493] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:39.419595 4443 raft_consensus.cc:515] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.421509 4443 leader_election.cc:290] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:39.423556 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcaae1b464378aabe7bc7f4bcc7fa" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:39.424225 3513 raft_consensus.cc:2468] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:39.432793 4433 tablet_bootstrap.cc:492] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:39.433234 4433 ts_tablet_manager.cc:1403] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.005s
I20260501 13:58:39.435891 4433 raft_consensus.cc:359] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.436574 4433 raft_consensus.cc:385] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.436837 4433 raft_consensus.cc:740] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.437495 4433 consensus_queue.cc:260] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.443547 4066 master_service.cc:954] ReplaceTablet: received request to replace tablet fb0b2d069e75401ba56839e072156ac7 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:39.451803 3757 leader_election.cc:304] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:39.453465 4433 ts_tablet_manager.cc:1434] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260501 13:58:39.455600 4433 tablet_bootstrap.cc:492] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:39.456463 4370 raft_consensus.cc:493] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:39.456887 4370 raft_consensus.cc:515] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.458706 4370 leader_election.cc:290] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:39.466171 4341 tablet_bootstrap.cc:492] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:39.471948 4341 tablet_bootstrap.cc:654] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.472484 4433 tablet_bootstrap.cc:654] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.474241 4443 raft_consensus.cc:493] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:39.474749 4443 raft_consensus.cc:515] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.485172 4341 tablet_bootstrap.cc:492] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:39.485610 4341 ts_tablet_manager.cc:1403] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20260501 13:58:39.488159 4341 raft_consensus.cc:359] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.488867 4341 raft_consensus.cc:385] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.489143 4341 raft_consensus.cc:740] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.489786 4341 consensus_queue.cc:260] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.501219 4443 leader_election.cc:290] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:39.502319 4433 tablet_bootstrap.cc:492] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:39.502769 4433 ts_tablet_manager.cc:1403] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.047s user 0.018s sys 0.013s
I20260501 13:58:39.504060 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcaae1b464378aabe7bc7f4bcc7fa" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:39.504662 3672 raft_consensus.cc:2468] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:39.505795 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6c11a5794f64e69a3f467800a7cd2ae" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:39.505648 4433 raft_consensus.cc:359] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.506358 4433 raft_consensus.cc:385] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.506693 4433 raft_consensus.cc:740] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.507257 4433 consensus_queue.cc:260] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.509523 4433 ts_tablet_manager.cc:1434] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260501 13:58:39.509738 4341 ts_tablet_manager.cc:1434] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260501 13:58:39.514381 4433 tablet_bootstrap.cc:492] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:39.515470 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41ac3120d65440829fe929b76cc41d43" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:39.516072 3674 raft_consensus.cc:2468] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:39.517128 3448 leader_election.cc:304] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:39.518445 4370 raft_consensus.cc:2804] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:39.518872 4370 raft_consensus.cc:493] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:39.519166 4370 raft_consensus.cc:3060] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.521066 4433 tablet_bootstrap.cc:654] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.522753 4066 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet fb0b2d069e75401ba56839e072156ac7
W20260501 13:58:39.522790 3756 leader_election.cc:343] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:39.524662 3491 tablet_service.cc:1558] Processing DeleteTablet for tablet fb0b2d069e75401ba56839e072156ac7 with delete_type TABLET_DATA_DELETED (replaced by tablet 65516a8e9190491587780880f6e1bbf9) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:39.525740 4066 catalog_manager.cc:6741] ReplaceTablet: tablet fb0b2d069e75401ba56839e072156ac7 deleted and replaced by tablet 65516a8e9190491587780880f6e1bbf9
I20260501 13:58:39.525975 4370 raft_consensus.cc:515] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.526415 3651 tablet_service.cc:1558] Processing DeleteTablet for tablet fb0b2d069e75401ba56839e072156ac7 with delete_type TABLET_DATA_DELETED (replaced by tablet 65516a8e9190491587780880f6e1bbf9) from {username='slave'} at 127.0.0.1:48686
W20260501 13:58:39.526348 4037 catalog_manager.cc:4982] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet fb0b2d069e75401ba56839e072156ac7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb0b2d069e75401ba56839e072156ac7 already in progress: creating tablet
W20260501 13:58:39.529824 4036 catalog_manager.cc:4982] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet fb0b2d069e75401ba56839e072156ac7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb0b2d069e75401ba56839e072156ac7 already in progress: creating tablet
I20260501 13:58:39.531759 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet fb0b2d069e75401ba56839e072156ac7 with delete_type TABLET_DATA_DELETED (replaced by tablet 65516a8e9190491587780880f6e1bbf9) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:39.532497 4359 tablet_replica.cc:333] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:39.532951 4370 leader_election.cc:290] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:39.533205 4359 raft_consensus.cc:2243] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:39.533648 4359 raft_consensus.cc:2272] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:39.534219 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41ac3120d65440829fe929b76cc41d43" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:39.534775 3674 raft_consensus.cc:3060] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.535701 4359 ts_tablet_manager.cc:1916] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:39.563922 4503 raft_consensus.cc:2804] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:39.564332 4503 raft_consensus.cc:493] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:39.564640 4503 raft_consensus.cc:3060] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.573783 4338 tablet_bootstrap.cc:492] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:39.574220 4338 ts_tablet_manager.cc:1403] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.250s user 0.014s sys 0.002s
I20260501 13:58:39.576808 4338 raft_consensus.cc:359] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.577415 4338 raft_consensus.cc:385] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.577664 4338 raft_consensus.cc:740] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.579130 4050 catalog_manager.cc:5671] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:39.587217 3491 tablet_service.cc:1511] Processing CreateTablet for tablet 65516a8e9190491587780880f6e1bbf9 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:39.588389 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65516a8e9190491587780880f6e1bbf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:39.595299 4066 catalog_manager.cc:5671] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9154033cea374a10a5225c53b8bcf122 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "9154033cea374a10a5225c53b8bcf122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:39.615880 4050 master_service.cc:954] ReplaceTablet: received request to replace tablet d6c11a5794f64e69a3f467800a7cd2ae from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:39.620920 3651 tablet_service.cc:1511] Processing CreateTablet for tablet 65516a8e9190491587780880f6e1bbf9 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:39.622052 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65516a8e9190491587780880f6e1bbf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:39.632917 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 65516a8e9190491587780880f6e1bbf9 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:39.634016 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65516a8e9190491587780880f6e1bbf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:39.634617 3674 raft_consensus.cc:2468] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:39.640825 4359 ts_tablet_manager.cc:1929] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:39.644034 4338 consensus_queue.cc:260] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.661854 4338 ts_tablet_manager.cc:1434] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.087s user 0.004s sys 0.001s
I20260501 13:58:39.675707 4338 tablet_bootstrap.cc:492] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:39.679905 4061 catalog_manager.cc:3502] renamed table default.table_06e2e16b792549e68d2e01ce3e3ce8b5 [id=9535ed975c6d458f97c9175b0a34f3ab] in HMS: new name default.table_995cfbea106b4656b4929310efb5017f
I20260501 13:58:39.650236 4341 tablet_bootstrap.cc:492] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
W20260501 13:58:39.659459 3757 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.129:44335, user_credentials={real_user=slave}} blocked reactor thread for 207759us
I20260501 13:58:39.689666 4359 log.cc:1199] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/fb0b2d069e75401ba56839e072156ac7
I20260501 13:58:39.690418 4359 ts_tablet_manager.cc:1950] T fb0b2d069e75401ba56839e072156ac7 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:39.692735 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet fb0b2d069e75401ba56839e072156ac7 (table default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]) successfully deleted
I20260501 13:58:39.701710 4050 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet d6c11a5794f64e69a3f467800a7cd2ae
I20260501 13:58:39.703333 4341 tablet_bootstrap.cc:654] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.707224 4050 catalog_manager.cc:6741] ReplaceTablet: tablet d6c11a5794f64e69a3f467800a7cd2ae deleted and replaced by tablet e12f37ec001949609d92e9118cbd9581
I20260501 13:58:39.708269 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet d6c11a5794f64e69a3f467800a7cd2ae with delete_type TABLET_DATA_DELETED (replaced by tablet e12f37ec001949609d92e9118cbd9581) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:39.708982 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet d6c11a5794f64e69a3f467800a7cd2ae with delete_type TABLET_DATA_DELETED (replaced by tablet e12f37ec001949609d92e9118cbd9581) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:39.709723 4358 tablet_replica.cc:333] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:39.710450 4358 raft_consensus.cc:2243] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:39.710904 4358 raft_consensus.cc:2272] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:39.711759 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41ac3120d65440829fe929b76cc41d43" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:39.712296 3822 raft_consensus.cc:2468] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:39.713456 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41ac3120d65440829fe929b76cc41d43" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:39.713935 3822 raft_consensus.cc:3060] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.715595 4358 ts_tablet_manager.cc:1916] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:39.719384 4382 consensus_queue.cc:1048] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:58:39.726444 3822 raft_consensus.cc:2468] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:39.726631 4066 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:39.729535 3448 leader_election.cc:304] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:39.731215 3492 tablet_service.cc:1558] Processing DeleteTablet for tablet d6c11a5794f64e69a3f467800a7cd2ae with delete_type TABLET_DATA_DELETED (replaced by tablet e12f37ec001949609d92e9118cbd9581) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:39.731707 4370 raft_consensus.cc:2804] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:39.732158 4370 raft_consensus.cc:697] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:39.732764 4357 tablet_replica.cc:333] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:39.734767 4357 raft_consensus.cc:2243] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:39.735206 4357 raft_consensus.cc:2272] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:39.736554 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6c11a5794f64e69a3f467800a7cd2ae" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:39.732879 4370 consensus_queue.cc:237] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.737370 4357 ts_tablet_manager.cc:1916] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:39.742172 4382 consensus_queue.cc:1048] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:58:39.752650 3757 leader_election.cc:343] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:39.756733 4433 tablet_bootstrap.cc:492] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:39.757261 4433 ts_tablet_manager.cc:1403] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.243s user 0.035s sys 0.077s
I20260501 13:58:39.759675 4358 ts_tablet_manager.cc:1929] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:39.760169 4358 log.cc:1199] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/d6c11a5794f64e69a3f467800a7cd2ae
I20260501 13:58:39.763042 4433 raft_consensus.cc:359] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.763952 4433 raft_consensus.cc:385] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.764258 4433 raft_consensus.cc:740] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.765050 4433 consensus_queue.cc:260] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.766229 4358 ts_tablet_manager.cc:1950] T d6c11a5794f64e69a3f467800a7cd2ae P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:39.774639 3757 leader_election.cc:304] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:39.775082 4433 ts_tablet_manager.cc:1434] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20260501 13:58:39.776067 4359 tablet_replica.cc:333] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:39.776755 4433 tablet_bootstrap.cc:492] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:39.782059 4433 tablet_bootstrap.cc:654] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.782542 4357 ts_tablet_manager.cc:1929] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:39.783151 4357 log.cc:1199] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/d6c11a5794f64e69a3f467800a7cd2ae
I20260501 13:58:39.783836 4357 ts_tablet_manager.cc:1950] T d6c11a5794f64e69a3f467800a7cd2ae P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:39.776888 4359 raft_consensus.cc:2243] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:39.786407 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet d6c11a5794f64e69a3f467800a7cd2ae (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
I20260501 13:58:39.790148 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet d6c11a5794f64e69a3f467800a7cd2ae (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
I20260501 13:58:39.799316 4433 tablet_bootstrap.cc:492] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:39.799880 4433 ts_tablet_manager.cc:1403] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.006s
I20260501 13:58:39.800261 4338 tablet_bootstrap.cc:654] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.802322 4433 raft_consensus.cc:359] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.803153 4433 raft_consensus.cc:385] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.803458 4433 raft_consensus.cc:740] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.804157 4433 consensus_queue.cc:260] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.805642 4498 raft_consensus.cc:2706] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20260501 13:58:39.806835 4359 raft_consensus.cc:2272] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:39.809615 4433 ts_tablet_manager.cc:1434] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260501 13:58:39.810593 4433 tablet_bootstrap.cc:492] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:39.820050 4433 tablet_bootstrap.cc:654] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:39.830485 4359 ts_tablet_manager.cc:1916] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:39.831003 4433 tablet_bootstrap.cc:492] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:39.831472 4433 ts_tablet_manager.cc:1403] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.009s
I20260501 13:58:39.833793 4433 raft_consensus.cc:359] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.834575 4433 raft_consensus.cc:385] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.834878 4433 raft_consensus.cc:740] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.835624 4433 consensus_queue.cc:260] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.839684 3651 tablet_service.cc:1558] Processing DeleteTablet for tablet d6c11a5794f64e69a3f467800a7cd2ae with delete_type TABLET_DATA_DELETED (replaced by tablet e12f37ec001949609d92e9118cbd9581) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:39.841375 4433 ts_tablet_manager.cc:1434] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.010s user 0.001s sys 0.003s
W20260501 13:58:39.841398 4036 catalog_manager.cc:4969] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet d6c11a5794f64e69a3f467800a7cd2ae because the tablet was not found. No further retry: Not found: Tablet not found: d6c11a5794f64e69a3f467800a7cd2ae
I20260501 13:58:39.650162 4503 raft_consensus.cc:515] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.863916 4503 leader_election.cc:290] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:39.864910 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcaae1b464378aabe7bc7f4bcc7fa" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:39.865454 3513 raft_consensus.cc:3060] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.867774 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcaae1b464378aabe7bc7f4bcc7fa" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:39.868345 3674 raft_consensus.cc:3060] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:39.872140 3513 raft_consensus.cc:2468] T addbcaae1b464378aabe7bc7f4bcc7fa P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:39.873688 3757 leader_election.cc:304] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:39.835886 4341 tablet_bootstrap.cc:492] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:39.877964 4341 ts_tablet_manager.cc:1403] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.228s user 0.013s sys 0.007s
I20260501 13:58:39.880481 4341 raft_consensus.cc:359] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.881161 4341 raft_consensus.cc:385] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.881413 4341 raft_consensus.cc:740] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.894892 3491 tablet_service.cc:1511] Processing CreateTablet for tablet e12f37ec001949609d92e9118cbd9581 (DEFAULT_TABLE table=default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:39.882030 4341 consensus_queue.cc:260] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.896021 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12f37ec001949609d92e9118cbd9581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:39.896983 3674 raft_consensus.cc:2468] T addbcaae1b464378aabe7bc7f4bcc7fa P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:39.897571 4498 raft_consensus.cc:2804] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:39.898034 4498 raft_consensus.cc:697] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:39.898823 4498 consensus_queue.cc:237] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:39.904918 4341 ts_tablet_manager.cc:1434] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.027s user 0.002s sys 0.004s
I20260501 13:58:39.924553 3651 tablet_service.cc:1511] Processing CreateTablet for tablet e12f37ec001949609d92e9118cbd9581 (DEFAULT_TABLE table=default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:39.925609 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12f37ec001949609d92e9118cbd9581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:39.955618 3802 tablet_service.cc:1511] Processing CreateTablet for tablet e12f37ec001949609d92e9118cbd9581 (DEFAULT_TABLE table=default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:39.956656 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12f37ec001949609d92e9118cbd9581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:39.961436 4309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 13:58:39.963579 4359 ts_tablet_manager.cc:1929] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:39.963946 4359 log.cc:1199] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/d6c11a5794f64e69a3f467800a7cd2ae
I20260501 13:58:39.964545 4359 ts_tablet_manager.cc:1950] T d6c11a5794f64e69a3f467800a7cd2ae P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:39.969116 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet d6c11a5794f64e69a3f467800a7cd2ae (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
I20260501 13:58:39.974768 4309 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 13:58:39.975422 4309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 13:58:39.980170 4338 tablet_bootstrap.cc:492] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:39.980618 4338 ts_tablet_manager.cc:1403] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.305s user 0.004s sys 0.008s
I20260501 13:58:39.979910 4066 catalog_manager.cc:5671] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9154033cea374a10a5225c53b8bcf122 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "9154033cea374a10a5225c53b8bcf122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:39.981221 4066 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:39.983989 4338 raft_consensus.cc:359] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.984629 4338 raft_consensus.cc:385] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:39.985888 4338 raft_consensus.cc:740] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:39.994907 4338 consensus_queue.cc:260] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:39.998427 4064 catalog_manager.cc:2174] created HMS catalog entry for table [id=9385850ae45c48ca955d499dd499a90b]
I20260501 13:58:39.999852 4433 tablet_bootstrap.cc:492] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:40.019680 4338 ts_tablet_manager.cc:1434] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.039s user 0.006s sys 0.001s
I20260501 13:58:40.025650 4338 tablet_bootstrap.cc:492] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
W20260501 13:58:40.026525 4054 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_82ebd50001e848a8b155dead3f78ea41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:40.038497 4050 master_service.cc:954] ReplaceTablet: received request to replace tablet fb0b2d069e75401ba56839e072156ac7 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:40.057821 4341 tablet_bootstrap.cc:492] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:40.063560 4341 tablet_bootstrap.cc:654] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:40.064007 4066 catalog_manager.cc:5671] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:40.065116 4064 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 67) took 2132 ms. Trace:
I20260501 13:58:40.065457 4064 rpcz_store.cc:276] 0501 13:58:37.932710 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:37.932880 (+ 170us) service_pool.cc:224] Handling call
0501 13:58:37.945256 (+ 12376us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5400030554. stack: 00007f4936204c83 00007f4936204b80 00007f4935d75574 000000000050e094 000000000050e029 00007f493d254470 00007f493d2f331a 00007f493f00746c 00007f493f102936 00007f493cd0f524 00007f493cd0f4b2 00007f493cd0f3e1 00007f493cd0f35c 00007f493cd0e612 00007f493afddbcc 00007f493afdd006
0501 13:58:38.815029 (+869773us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:38.816666 (+ 1637us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:39.998386 (+1181720us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:40.064404 (+ 66018us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:40.064778 (+ 374us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:40.065051 (+ 273us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:40.000119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:40.000505 (+ 386us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:40.000529 (+ 24us) write_op.cc:435] Acquired schema lock
0501 13:58:40.000543 (+ 14us) tablet.cc:662] Decoding operations
0501 13:58:40.000810 (+ 267us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:40.000849 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:40.000870 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:40.001046 (+ 176us) tablet.cc:701] Row locks acquired
0501 13:58:40.001059 (+ 13us) write_op.cc:260] PREPARE: finished
0501 13:58:40.001173 (+ 114us) write_op.cc:270] Start()
0501 13:58:40.001248 (+ 75us) write_op.cc:276] Timestamp: P: 1777643920001140 usec, L: 0
0501 13:58:40.001263 (+ 15us) op_driver.cc:348] REPLICATION: starting
0501 13:58:40.001800 (+ 537us) log.cc:844] Serialized 433 byte log entry
0501 13:58:40.012973 (+ 11173us) spinlock_profiling.cc:243] Waited 9.93 ms on lock 0x7b58000d01d0. stack: 00007f4936204c83 00007f4936204b80 00007f4935d75574 000000000050e094 000000000050e029 00007f493d254470 00007f493d24481c 00007f493d24b133 00007f493d24b0b9 00007f493d24b049 00007f493d24b011 00007f493d24a1dd 00000000005b3bc4 00000000005b39f9 00007f4936236ab5 00007f4936238e01
0501 13:58:40.056202 (+ 43229us) op_driver.cc:464] REPLICATION: finished
0501 13:58:40.061996 (+ 5794us) write_op.cc:301] APPLY: starting
0501 13:58:40.062065 (+ 69us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:40.062168 (+ 103us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:40.062188 (+ 20us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:40.062879 (+ 691us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:40.063031 (+ 152us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:40.063079 (+ 48us) write_op.cc:312] APPLY: finished
0501 13:58:40.063285 (+ 206us) log.cc:844] Serialized 29 byte log entry
0501 13:58:40.063455 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:40.063528 (+ 73us) write_op.cc:454] Released schema lock
0501 13:58:40.063561 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":862286,"Hive Metastore.run_cpu_time_us":593,"Hive Metastore.run_wall_time_us":318696,"spinlock_wait_cycles":26307456,"child_traces":[["op",{"apply.queue_time_us":5588,"apply.run_cpu_time_us":1579,"apply.run_wall_time_us":1787,"num_ops":2,"prepare.queue_time_us":202,"prepare.run_cpu_time_us":1974,"prepare.run_wall_time_us":2287,"raft.queue_time_us":203,"raft.run_cpu_time_us":1144,"raft.run_wall_time_us":10830,"replication_time_us":54808,"spinlock_wait_cycles":22850304}]]}
I20260501 13:58:40.070040 4370 raft_consensus.cc:493] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:40.070550 4370 raft_consensus.cc:515] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.072283 4370 leader_election.cc:290] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:40.074856 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65516a8e9190491587780880f6e1bbf9" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:40.076009 4338 tablet_bootstrap.cc:654] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:40.076056 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65516a8e9190491587780880f6e1bbf9" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:40.076679 3821 raft_consensus.cc:2468] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
W20260501 13:58:40.078310 3448 leader_election.cc:343] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:40.078990 3448 leader_election.cc:304] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:40.080144 4370 raft_consensus.cc:2804] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:40.080503 4370 raft_consensus.cc:493] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:40.080801 4370 raft_consensus.cc:3060] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:40.084029 4341 tablet_bootstrap.cc:492] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:40.085357 4341 ts_tablet_manager.cc:1403] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20260501 13:58:40.087837 4341 raft_consensus.cc:359] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.088478 4341 raft_consensus.cc:385] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:40.088744 4341 raft_consensus.cc:740] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:40.089363 4341 consensus_queue.cc:260] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.090099 4370 raft_consensus.cc:515] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.097043 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65516a8e9190491587780880f6e1bbf9" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
W20260501 13:58:40.098413 3448 leader_election.cc:343] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:40.100346 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65516a8e9190491587780880f6e1bbf9" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:40.100891 3821 raft_consensus.cc:3060] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:40.101869 4341 ts_tablet_manager.cc:1434] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260501 13:58:40.104373 4433 tablet_bootstrap.cc:654] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:40.105437 4490 raft_consensus.cc:493] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:40.110358 3821 raft_consensus.cc:2468] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:40.105923 4490 raft_consensus.cc:515] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.111497 3448 leader_election.cc:304] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:40.112080 4490 leader_election.cc:290] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:40.113163 4490 raft_consensus.cc:2804] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:40.113966 4490 raft_consensus.cc:697] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:40.115768 4490 consensus_queue.cc:237] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.121820 4066 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet e12f37ec001949609d92e9118cbd9581 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]). Expected version 1 got 0
I20260501 13:58:40.116539 4370 leader_election.cc:290] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:40.131022 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "776eacf49dbd4c2f9a8dfa1ae2181c96" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:40.133431 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "776eacf49dbd4c2f9a8dfa1ae2181c96" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:40.134101 3821 raft_consensus.cc:2468] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:40.137877 3448 leader_election.cc:304] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
W20260501 13:58:40.138844 3448 leader_election.cc:343] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
W20260501 13:58:40.139338 4066 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet e12f37ec001949609d92e9118cbd9581 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e12f37ec001949609d92e9118cbd9581 consensus state has no leader
I20260501 13:58:40.140131 4370 raft_consensus.cc:2804] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:40.140559 4370 raft_consensus.cc:493] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:40.140970 4370 raft_consensus.cc:3060] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:40.143373 4338 tablet_bootstrap.cc:492] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:40.143836 4338 ts_tablet_manager.cc:1403] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.118s user 0.010s sys 0.000s
I20260501 13:58:40.143940 4525 consensus_queue.cc:1048] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260501 13:58:40.148883 4064 master_service.cc:954] ReplaceTablet: received request to replace tablet ba69864093504babbb00fb6170774778 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:40.150107 3651 tablet_service.cc:1558] Processing DeleteTablet for tablet fb0b2d069e75401ba56839e072156ac7 with delete_type TABLET_DATA_DELETED (replaced by tablet 65516a8e9190491587780880f6e1bbf9) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:40.151176 4358 tablet_replica.cc:333] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:40.152007 4358 raft_consensus.cc:2243] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:40.152463 4358 raft_consensus.cc:2272] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:40.155786 4358 ts_tablet_manager.cc:1916] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:40.174801 4370 raft_consensus.cc:515] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.175877 4433 tablet_bootstrap.cc:492] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:40.176308 4433 ts_tablet_manager.cc:1403] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.177s user 0.009s sys 0.004s
I20260501 13:58:40.177876 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "776eacf49dbd4c2f9a8dfa1ae2181c96" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:40.179015 4490 raft_consensus.cc:493] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 13:58:40.179770 3448 leader_election.cc:343] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:40.179507 4490 raft_consensus.cc:515] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
W20260501 13:58:40.183825 4036 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet e12f37ec001949609d92e9118cbd9581 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e12f37ec001949609d92e9118cbd9581 consensus state has no leader
I20260501 13:58:40.183666 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e76ac9fdeed442b94510860cd439b24" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:40.186129 3491 tablet_service.cc:1558] Processing DeleteTablet for tablet fb0b2d069e75401ba56839e072156ac7 with delete_type TABLET_DATA_DELETED (replaced by tablet 65516a8e9190491587780880f6e1bbf9) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:40.186885 4357 tablet_replica.cc:333] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:40.187620 4357 raft_consensus.cc:2243] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:40.188055 4357 raft_consensus.cc:2272] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:40.188086 3674 raft_consensus.cc:1275] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Refusing update from remote peer 9154033cea374a10a5225c53b8bcf122: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:58:40.189849 4357 ts_tablet_manager.cc:1916] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:40.190446 3821 raft_consensus.cc:1275] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Refusing update from remote peer 9154033cea374a10a5225c53b8bcf122: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260501 13:58:40.191632 3448 leader_election.cc:343] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:40.192304 4498 raft_consensus.cc:493] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:40.192982 4490 leader_election.cc:290] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:40.192766 4498 raft_consensus.cc:515] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.193449 4490 consensus_queue.cc:1048] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:40.194766 4490 consensus_queue.cc:1048] T 533eb8337c414aea98ea95d7bf507b1e P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:40.195933 4490 consensus_queue.cc:1048] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:40.197167 4498 leader_election.cc:290] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:40.198665 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e76ac9fdeed442b94510860cd439b24" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
W20260501 13:58:40.199882 3756 leader_election.cc:343] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:40.201368 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e76ac9fdeed442b94510860cd439b24" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:40.202111 3821 raft_consensus.cc:2468] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:40.204535 3491 tablet_service.cc:1511] Processing CreateTablet for tablet eb5de60e4d1041f29ef1ecb2d0181a26 (DEFAULT_TABLE table=default.table_59dc65b11ade4ee3acb91116f7a60585 [id=9385850ae45c48ca955d499dd499a90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:40.205543 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5de60e4d1041f29ef1ecb2d0181a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:40.205752 4050 catalog_manager.cc:5671] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9154033cea374a10a5225c53b8bcf122 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "9154033cea374a10a5225c53b8bcf122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:40.212487 3651 tablet_service.cc:1511] Processing CreateTablet for tablet eb5de60e4d1041f29ef1ecb2d0181a26 (DEFAULT_TABLE table=default.table_59dc65b11ade4ee3acb91116f7a60585 [id=9385850ae45c48ca955d499dd499a90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:40.213546 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5de60e4d1041f29ef1ecb2d0181a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:40.215608 3802 tablet_service.cc:1511] Processing CreateTablet for tablet eb5de60e4d1041f29ef1ecb2d0181a26 (DEFAULT_TABLE table=default.table_59dc65b11ade4ee3acb91116f7a60585 [id=9385850ae45c48ca955d499dd499a90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:40.216580 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5de60e4d1041f29ef1ecb2d0181a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:40.223649 4358 ts_tablet_manager.cc:1929] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:40.178768 4433 raft_consensus.cc:359] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.226711 4433 raft_consensus.cc:385] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:40.227073 4433 raft_consensus.cc:740] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:40.227941 4433 consensus_queue.cc:260] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.229357 4338 raft_consensus.cc:359] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.229993 4338 raft_consensus.cc:385] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:40.230239 4338 raft_consensus.cc:740] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:40.230425 4433 ts_tablet_manager.cc:1434] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20260501 13:58:40.230967 4338 consensus_queue.cc:260] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.235563 4338 ts_tablet_manager.cc:1434] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.091s user 0.006s sys 0.000s
I20260501 13:58:40.224032 4358 log.cc:1199] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/fb0b2d069e75401ba56839e072156ac7
I20260501 13:58:40.236793 4358 ts_tablet_manager.cc:1950] T fb0b2d069e75401ba56839e072156ac7 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:40.239274 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet fb0b2d069e75401ba56839e072156ac7 (table default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]) successfully deleted
I20260501 13:58:40.245813 4370 leader_election.cc:290] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:40.248092 3448 leader_election.cc:304] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:40.249163 4524 raft_consensus.cc:2804] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:40.249517 4524 raft_consensus.cc:493] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:40.249830 4524 raft_consensus.cc:3060] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:40.254611 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e76ac9fdeed442b94510860cd439b24" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:40.265039 4524 raft_consensus.cc:515] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.259137 4338 tablet_bootstrap.cc:492] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:40.269889 4050 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:40.272305 4338 tablet_bootstrap.cc:654] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:40.276026 4066 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_8f8971cc622d4dca8d21a3656e765390"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:40.287215 3513 raft_consensus.cc:2393] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c239ccbdc554619a237d78e958ffbc2 in current term 1: Already voted for candidate 9154033cea374a10a5225c53b8bcf122 in this term.
I20260501 13:58:40.289580 4524 leader_election.cc:290] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:40.293182 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "776eacf49dbd4c2f9a8dfa1ae2181c96" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:40.293702 3822 raft_consensus.cc:3060] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:40.299178 4382 consensus_queue.cc:1048] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:40.302704 3757 leader_election.cc:304] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:40.303406 4498 raft_consensus.cc:3060] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:40.310351 4338 tablet_bootstrap.cc:492] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:40.310850 4338 ts_tablet_manager.cc:1403] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.052s user 0.013s sys 0.000s
I20260501 13:58:40.319077 4498 raft_consensus.cc:2749] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:40.320534 4050 master_service.cc:954] ReplaceTablet: received request to replace tablet a9ecb5b51ae04f32a4f07f828d563fa4 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:40.322434 4341 tablet_bootstrap.cc:492] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:40.323298 4433 tablet_bootstrap.cc:492] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:40.328305 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e76ac9fdeed442b94510860cd439b24" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:40.328492 4338 raft_consensus.cc:359] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.329008 3672 raft_consensus.cc:3060] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:40.329475 4341 tablet_bootstrap.cc:654] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:40.331261 4357 ts_tablet_manager.cc:1929] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:40.332093 4357 log.cc:1199] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/fb0b2d069e75401ba56839e072156ac7
I20260501 13:58:40.332707 4357 ts_tablet_manager.cc:1950] T fb0b2d069e75401ba56839e072156ac7 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:40.329139 4338 raft_consensus.cc:385] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:40.335806 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet fb0b2d069e75401ba56839e072156ac7 (table default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]) successfully deleted
I20260501 13:58:40.336227 4338 raft_consensus.cc:740] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:40.337507 4338 consensus_queue.cc:260] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.339073 4487 consensus_queue.cc:1048] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:40.339903 4338 ts_tablet_manager.cc:1434] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20260501 13:58:40.350188 4052 catalog_manager.cc:2174] created HMS catalog entry for table [id=eac59b190539443d88199a65abe479b9]
I20260501 13:58:40.355839 3820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e76ac9fdeed442b94510860cd439b24" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:40.362792 4341 tablet_bootstrap.cc:492] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:40.363233 4341 ts_tablet_manager.cc:1403] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.006s
I20260501 13:58:40.364135 4187 catalog_manager.cc:2772] Deleting table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87] in response to Hive Metastore notification log event 26
I20260501 13:58:40.370008 4341 raft_consensus.cc:359] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.370748 4341 raft_consensus.cc:385] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:40.371006 4341 raft_consensus.cc:740] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:40.371632 4341 consensus_queue.cc:260] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.373373 3820 raft_consensus.cc:2468] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:40.379169 4524 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:40.379657 4524 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.381536 4524 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:40.389937 4052 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 64) took 2902 ms. Trace:
I20260501 13:58:40.399981 4542 tablet.cc:1724] T 41ac3120d65440829fe929b76cc41d43 P 9154033cea374a10a5225c53b8bcf122: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 13:58:40.401540 3448 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.130:45857, user_credentials={real_user=slave}} blocked reactor thread for 46453.6us
I20260501 13:58:40.398732 4052 rpcz_store.cc:276] 0501 13:58:37.488079 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:37.581942 (+ 93863us) service_pool.cc:224] Handling call
0501 13:58:38.810926 (+1228984us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:38.818709 (+ 7783us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:40.350131 (+1531422us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:40.389236 (+ 39105us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:40.389565 (+ 329us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:40.389865 (+ 300us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:40.352612 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:40.353003 (+ 391us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:40.353028 (+ 25us) write_op.cc:435] Acquired schema lock
0501 13:58:40.353039 (+ 11us) tablet.cc:662] Decoding operations
0501 13:58:40.353293 (+ 254us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:40.353330 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:40.353351 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:40.353534 (+ 183us) tablet.cc:701] Row locks acquired
0501 13:58:40.353546 (+ 12us) write_op.cc:260] PREPARE: finished
0501 13:58:40.353659 (+ 113us) write_op.cc:270] Start()
0501 13:58:40.353740 (+ 81us) write_op.cc:276] Timestamp: P: 1777643920353624 usec, L: 0
0501 13:58:40.353755 (+ 15us) op_driver.cc:348] REPLICATION: starting
0501 13:58:40.354247 (+ 492us) log.cc:844] Serialized 433 byte log entry
0501 13:58:40.386296 (+ 32049us) op_driver.cc:464] REPLICATION: finished
0501 13:58:40.386753 (+ 457us) write_op.cc:301] APPLY: starting
0501 13:58:40.386810 (+ 57us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:40.386911 (+ 101us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:40.387003 (+ 92us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:40.387409 (+ 406us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:40.387542 (+ 133us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:40.387585 (+ 43us) write_op.cc:312] APPLY: finished
0501 13:58:40.387777 (+ 192us) log.cc:844] Serialized 29 byte log entry
0501 13:58:40.387939 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:40.388011 (+ 72us) write_op.cc:454] Released schema lock
0501 13:58:40.388042 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0501 13:58:40.388268 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:40.388290 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1204517,"Hive Metastore.run_cpu_time_us":691,"Hive Metastore.run_wall_time_us":325985,"child_traces":[["op",{"apply.queue_time_us":289,"apply.run_cpu_time_us":1481,"apply.run_wall_time_us":1480,"num_ops":2,"prepare.queue_time_us":476,"prepare.run_cpu_time_us":1985,"prepare.run_wall_time_us":1986,"raft.queue_time_us":4148,"raft.run_cpu_time_us":640,"raft.run_wall_time_us":639,"replication_time_us":32442}]]}
I20260501 13:58:40.410422 4341 ts_tablet_manager.cc:1434] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20260501 13:58:40.414812 4338 tablet_bootstrap.cc:492] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:40.419610 4050 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:40.431260 3822 raft_consensus.cc:2468] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:40.435220 3448 leader_election.cc:304] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:40.438877 3672 raft_consensus.cc:2468] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:40.440351 4524 raft_consensus.cc:2804] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:40.440831 4524 raft_consensus.cc:697] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:40.441592 4524 consensus_queue.cc:237] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.414350 4433 tablet_bootstrap.cc:654] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:40.453945 4187 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet e12f37ec001949609d92e9118cbd9581
W20260501 13:58:40.455852 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:58:40.456296 3491 tablet_service.cc:1558] Processing DeleteTablet for tablet e12f37ec001949609d92e9118cbd9581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:40 UTC) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:40.456632 3448 leader_election.cc:304] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:40.456980 4357 tablet_replica.cc:333] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:40.457679 4357 raft_consensus.cc:2243] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:40.463927 4490 raft_consensus.cc:2804] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:40.464665 4490 raft_consensus.cc:697] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:40.465461 4490 consensus_queue.cc:237] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.468608 4357 raft_consensus.cc:2272] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:40.474738 4357 ts_tablet_manager.cc:1916] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:58:40.475485 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:58:40.475857 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:40.476436 4525 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:40.490392 4338 tablet_bootstrap.cc:654] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:40.496634 3651 tablet_service.cc:1558] Processing DeleteTablet for tablet e12f37ec001949609d92e9118cbd9581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:40 UTC) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:40.497449 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet e12f37ec001949609d92e9118cbd9581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:40 UTC) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:40.516505 4549 tablet_replica.cc:333] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:40.517519 4549 raft_consensus.cc:2243] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:40.518082 4549 raft_consensus.cc:2272] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:40.522295 4549 ts_tablet_manager.cc:1916] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:58:40.525878 4036 catalog_manager.cc:4982] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): delete failed for tablet e12f37ec001949609d92e9118cbd9581 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e12f37ec001949609d92e9118cbd9581 already in progress: creating tablet
I20260501 13:58:40.532500 4052 master_service.cc:954] ReplaceTablet: received request to replace tablet e12f37ec001949609d92e9118cbd9581 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:40.542054 4050 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
I20260501 13:58:40.543916 4548 tablet.cc:1724] T 41ac3120d65440829fe929b76cc41d43 P 6af0cedc7f3c47299b93e6e92d8e02d1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:40.548717 4549 ts_tablet_manager.cc:1929] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:40.551605 4549 log.cc:1199] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/e12f37ec001949609d92e9118cbd9581
I20260501 13:58:40.552366 4549 ts_tablet_manager.cc:1950] T e12f37ec001949609d92e9118cbd9581 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:40.505290 4543 tablet.cc:1724] T 41ac3120d65440829fe929b76cc41d43 P 5c239ccbdc554619a237d78e958ffbc2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:40.567284 4055 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]
I20260501 13:58:40.573541 4052 master_service.cc:954] ReplaceTablet: received request to replace tablet 65516a8e9190491587780880f6e1bbf9 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:40.574889 4338 tablet_bootstrap.cc:492] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:40.575346 4338 ts_tablet_manager.cc:1403] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.235s user 0.006s sys 0.004s
I20260501 13:58:40.577836 4338 raft_consensus.cc:359] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.583523 4338 raft_consensus.cc:385] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:40.583847 4338 raft_consensus.cc:740] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:40.585992 4338 consensus_queue.cc:260] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.599414 4338 ts_tablet_manager.cc:1434] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260501 13:58:40.599452 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet e12f37ec001949609d92e9118cbd9581 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
I20260501 13:58:40.607686 4338 tablet_bootstrap.cc:492] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:40.609009 4052 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet 65516a8e9190491587780880f6e1bbf9
I20260501 13:58:40.611039 3491 tablet_service.cc:1558] Processing DeleteTablet for tablet 65516a8e9190491587780880f6e1bbf9 with delete_type TABLET_DATA_DELETED (replaced by tablet 322b71262eb44077bbe0cfcebc108882) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:40.612362 4052 catalog_manager.cc:6741] ReplaceTablet: tablet 65516a8e9190491587780880f6e1bbf9 deleted and replaced by tablet 322b71262eb44077bbe0cfcebc108882
I20260501 13:58:40.619401 4338 tablet_bootstrap.cc:654] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:40.637609 4498 raft_consensus.cc:493] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:40.638121 4498 raft_consensus.cc:515] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.640017 4498 leader_election.cc:290] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:40.642303 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb5de60e4d1041f29ef1ecb2d0181a26" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:40.643131 4357 ts_tablet_manager.cc:1929] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:40.643584 4357 log.cc:1199] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/e12f37ec001949609d92e9118cbd9581
I20260501 13:58:40.643909 4338 tablet_bootstrap.cc:492] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
W20260501 13:58:40.643823 3756 leader_election.cc:343] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:40.644446 4338 ts_tablet_manager.cc:1403] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.000s
I20260501 13:58:40.648061 3651 tablet_service.cc:1558] Processing DeleteTablet for tablet 65516a8e9190491587780880f6e1bbf9 with delete_type TABLET_DATA_DELETED (replaced by tablet 322b71262eb44077bbe0cfcebc108882) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:40.648556 4338 raft_consensus.cc:359] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.649163 4338 raft_consensus.cc:385] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:40.649237 4358 tablet_replica.cc:333] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:40.649435 4338 raft_consensus.cc:740] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:40.649959 4358 raft_consensus.cc:2243] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:40.650447 4358 raft_consensus.cc:2272] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:40.652242 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 65516a8e9190491587780880f6e1bbf9 with delete_type TABLET_DATA_DELETED (replaced by tablet 322b71262eb44077bbe0cfcebc108882) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:40.652549 4358 ts_tablet_manager.cc:1916] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:40.652990 4549 tablet_replica.cc:333] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:40.653798 4549 raft_consensus.cc:2243] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:40.654254 4549 raft_consensus.cc:2272] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:40.656596 4549 ts_tablet_manager.cc:1916] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:40.662750 4338 consensus_queue.cc:260] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.673286 4338 ts_tablet_manager.cc:1434] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260501 13:58:40.678699 4338 tablet_bootstrap.cc:492] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:40.686590 4358 ts_tablet_manager.cc:1929] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:40.696807 4054 catalog_manager.cc:2174] created HMS catalog entry for table [id=18405c74f9e14a2980c453b17acd0ac9]
I20260501 13:58:40.696924 4358 log.cc:1199] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/65516a8e9190491587780880f6e1bbf9
I20260501 13:58:40.697751 4338 tablet_bootstrap.cc:654] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:40.644579 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb5de60e4d1041f29ef1ecb2d0181a26" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:40.698905 4549 ts_tablet_manager.cc:1929] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:40.646684 4433 tablet_bootstrap.cc:492] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:40.699262 4549 log.cc:1199] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/65516a8e9190491587780880f6e1bbf9
I20260501 13:58:40.699424 4433 ts_tablet_manager.cc:1403] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.376s user 0.015s sys 0.014s
I20260501 13:58:40.699018 4357 ts_tablet_manager.cc:1950] T e12f37ec001949609d92e9118cbd9581 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:40.699923 4549 ts_tablet_manager.cc:1950] T 65516a8e9190491587780880f6e1bbf9 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:40.700402 4358 ts_tablet_manager.cc:1950] T 65516a8e9190491587780880f6e1bbf9 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:40.701782 4433 raft_consensus.cc:359] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.702435 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 65516a8e9190491587780880f6e1bbf9 (table default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]) successfully deleted
I20260501 13:58:40.703081 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 65516a8e9190491587780880f6e1bbf9 (table default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]) successfully deleted
W20260501 13:58:40.703075 3757 leader_election.cc:343] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Illegal state: must be running to vote when last-logged opid is not known
I20260501 13:58:40.703462 3757 leader_election.cc:304] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:40.704365 4498 raft_consensus.cc:2749] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:40.702400 4433 raft_consensus.cc:385] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:40.705555 4433 raft_consensus.cc:740] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:40.706416 4433 consensus_queue.cc:260] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.708884 4338 tablet_bootstrap.cc:492] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:40.709337 4338 ts_tablet_manager.cc:1403] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.000s
I20260501 13:58:40.711930 4338 raft_consensus.cc:359] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.712603 4338 raft_consensus.cc:385] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:40.712849 4338 raft_consensus.cc:740] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:40.718228 4498 consensus_queue.cc:1048] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:40.718788 4338 consensus_queue.cc:260] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.725425 4487 raft_consensus.cc:493] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:40.727698 4433 ts_tablet_manager.cc:1434] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260501 13:58:40.725893 4487 raft_consensus.cc:515] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:40.730726 4487 leader_election.cc:290] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:40.732836 4487 raft_consensus.cc:493] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:40.732777 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "776eacf49dbd4c2f9a8dfa1ae2181c96" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:40.733371 4487 raft_consensus.cc:515] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.738895 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb5de60e4d1041f29ef1ecb2d0181a26" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:40.739430 4487 leader_election.cc:290] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:40.739526 3822 raft_consensus.cc:2468] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:40.740630 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb5de60e4d1041f29ef1ecb2d0181a26" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:40.740669 3608 leader_election.cc:304] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:40.741228 3513 raft_consensus.cc:2468] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:40.741457 4487 raft_consensus.cc:2804] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:40.741801 4487 raft_consensus.cc:493] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:40.742106 4487 raft_consensus.cc:3060] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:40.745474 4338 ts_tablet_manager.cc:1434] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20260501 13:58:40.746963 3820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "776eacf49dbd4c2f9a8dfa1ae2181c96" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:40.747732 3820 raft_consensus.cc:2393] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in current term 1: Already voted for candidate 9154033cea374a10a5225c53b8bcf122 in this term.
I20260501 13:58:40.749927 3608 leader_election.cc:304] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1; no voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:40.750974 4421 raft_consensus.cc:3060] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:40.751827 4054 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 73) took 2179 ms. Trace:
I20260501 13:58:40.757555 4487 raft_consensus.cc:515] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.759168 4498 consensus_queue.cc:1048] T addbcaae1b464378aabe7bc7f4bcc7fa P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:40.752159 4054 rpcz_store.cc:276] 0501 13:58:38.572809 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:38.592142 (+ 19333us) service_pool.cc:224] Handling call
0501 13:58:40.026947 (+1434805us) catalog_manager.cc:2101] Acquired catalog manager lock
0501 13:58:40.028778 (+ 1831us) catalog_manager.cc:2147] Created new table and tablet info
0501 13:58:40.696763 (+667985us) catalog_manager.cc:2173] Created new table in HMS catalog
0501 13:58:40.751122 (+ 54359us) catalog_manager.cc:2207] Wrote table and tablets to system table
0501 13:58:40.751441 (+ 319us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0501 13:58:40.751755 (+ 314us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0501 13:58:40.715152 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 13:58:40.715562 (+ 410us) write_op.cc:432] Acquiring schema lock in shared mode
0501 13:58:40.715583 (+ 21us) write_op.cc:435] Acquired schema lock
0501 13:58:40.715595 (+ 12us) tablet.cc:662] Decoding operations
0501 13:58:40.715843 (+ 248us) write_op.cc:620] Acquiring the partition lock for write op
0501 13:58:40.715875 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0501 13:58:40.715895 (+ 20us) tablet.cc:685] Acquiring locks for 2 operations
0501 13:58:40.716067 (+ 172us) tablet.cc:701] Row locks acquired
0501 13:58:40.716079 (+ 12us) write_op.cc:260] PREPARE: finished
0501 13:58:40.716239 (+ 160us) write_op.cc:270] Start()
0501 13:58:40.716333 (+ 94us) write_op.cc:276] Timestamp: P: 1777643920716203 usec, L: 0
0501 13:58:40.716348 (+ 15us) op_driver.cc:348] REPLICATION: starting
0501 13:58:40.716809 (+ 461us) log.cc:844] Serialized 433 byte log entry
0501 13:58:40.739424 (+ 22615us) op_driver.cc:464] REPLICATION: finished
0501 13:58:40.741614 (+ 2190us) write_op.cc:301] APPLY: starting
0501 13:58:40.741666 (+ 52us) tablet.cc:1367] starting BulkCheckPresence
0501 13:58:40.741758 (+ 92us) tablet.cc:1370] finished BulkCheckPresence
0501 13:58:40.741830 (+ 72us) tablet.cc:1372] starting ApplyRowOperation cycle
0501 13:58:40.742215 (+ 385us) tablet.cc:1383] finished ApplyRowOperation cycle
0501 13:58:40.742346 (+ 131us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 13:58:40.742385 (+ 39us) write_op.cc:312] APPLY: finished
0501 13:58:40.750420 (+ 8035us) log.cc:844] Serialized 29 byte log entry
0501 13:58:40.750626 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0501 13:58:40.750704 (+ 78us) write_op.cc:454] Released schema lock
0501 13:58:40.750738 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":537496,"Hive Metastore.run_cpu_time_us":613,"Hive Metastore.run_wall_time_us":129749,"child_traces":[["op",{"apply.queue_time_us":2040,"apply.run_cpu_time_us":1563,"apply.run_wall_time_us":9321,"mutex_wait_us":11304,"num_ops":2,"prepare.queue_time_us":16723,"prepare.run_cpu_time_us":1882,"prepare.run_wall_time_us":2945,"raft.queue_time_us":74,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":15,"replication_time_us":22992,"threads_started":1}]]}
I20260501 13:58:40.764691 4421 raft_consensus.cc:2749] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:40.767756 3820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb5de60e4d1041f29ef1ecb2d0181a26" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:40.768392 3820 raft_consensus.cc:3060] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:40.779276 3491 tablet_service.cc:1511] Processing CreateTablet for tablet 70c23102f4004fcc83ce8e492b4f1d54 (DEFAULT_TABLE table=default.table_8be6ed54e88344f7b4be783af8fa48da [id=eac59b190539443d88199a65abe479b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:40.780458 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70c23102f4004fcc83ce8e492b4f1d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:40.781292 4487 leader_election.cc:290] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:40.781461 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb5de60e4d1041f29ef1ecb2d0181a26" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:40.781978 3515 raft_consensus.cc:3060] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:40.782286 3820 raft_consensus.cc:2468] T eb5de60e4d1041f29ef1ecb2d0181a26 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:40.783670 3608 leader_election.cc:304] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:40.784651 4487 raft_consensus.cc:2804] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:40.785117 4487 raft_consensus.cc:697] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:40.785933 4487 consensus_queue.cc:237] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.787737 3515 raft_consensus.cc:2468] T eb5de60e4d1041f29ef1ecb2d0181a26 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:40.796245 4052 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]). Expected version 1 got 0
W20260501 13:58:40.803500 4036 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.190:38561, user_credentials={real_user=slave}} blocked reactor thread for 64925.4us
I20260501 13:58:40.805608 3651 tablet_service.cc:1511] Processing CreateTablet for tablet 70c23102f4004fcc83ce8e492b4f1d54 (DEFAULT_TABLE table=default.table_8be6ed54e88344f7b4be783af8fa48da [id=eac59b190539443d88199a65abe479b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:40.806864 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70c23102f4004fcc83ce8e492b4f1d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:40.817052 4054 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_96b1c2cf6e7c4d82a665a14d3decdab3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:58:40.843945 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 70c23102f4004fcc83ce8e492b4f1d54 (DEFAULT_TABLE table=default.table_8be6ed54e88344f7b4be783af8fa48da [id=eac59b190539443d88199a65abe479b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:40.845002 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70c23102f4004fcc83ce8e492b4f1d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:40.859453 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet e12f37ec001949609d92e9118cbd9581 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
I20260501 13:58:40.859772 4357 tablet_replica.cc:333] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:40.860481 4357 raft_consensus.cc:2243] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:40.861147 4357 pending_rounds.cc:70] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: Trying to abort 1 pending ops.
I20260501 13:58:40.861384 4357 pending_rounds.cc:77] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229496812142592 op_type: NO_OP noop_request { }
I20260501 13:58:40.861732 4341 tablet_bootstrap.cc:492] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:40.861725 4357 raft_consensus.cc:2889] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:40.862033 4357 raft_consensus.cc:2272] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:40.868116 4341 tablet_bootstrap.cc:654] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:40.882958 4341 tablet_bootstrap.cc:492] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:40.883409 4341 ts_tablet_manager.cc:1403] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.006s
I20260501 13:58:40.885816 4341 raft_consensus.cc:359] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.886493 4341 raft_consensus.cc:385] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:40.886729 4341 raft_consensus.cc:740] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:40.890645 4341 consensus_queue.cc:260] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:40.895032 4341 ts_tablet_manager.cc:1434] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260501 13:58:40.914166 4433 tablet_bootstrap.cc:492] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:40.920389 4064 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef] alter complete (version 1)
I20260501 13:58:40.921648 4064 master_service.cc:954] ReplaceTablet: received request to replace tablet 1505e4bc749b48a88008d732881ff148 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:40.929620 3492 tablet_service.cc:1558] Processing DeleteTablet for tablet e12f37ec001949609d92e9118cbd9581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:40 UTC) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:40.932770 4357 ts_tablet_manager.cc:1916] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:40.939761 4433 tablet_bootstrap.cc:654] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:40.945441 4052 master_service.cc:954] ReplaceTablet: received request to replace tablet 80a2f6fab87a4ad980c7816f34f41aff from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:40.969983 4063 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35438 (request call id 58) took 4203 ms. Trace:
I20260501 13:58:40.970395 4063 rpcz_store.cc:276] 0501 13:58:36.766668 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:36.770584 (+ 3916us) service_pool.cc:224] Handling call
0501 13:58:38.807102 (+2036518us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:40.969909 (+2162807us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":7051,"Hive Metastore.run_cpu_time_us":393,"Hive Metastore.run_wall_time_us":274555}
I20260501 13:58:40.974813 4061 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35438 (request call id 59) took 4196 ms. Trace:
I20260501 13:58:40.975142 4061 rpcz_store.cc:276] 0501 13:58:36.778065 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:36.778191 (+ 126us) service_pool.cc:224] Handling call
0501 13:58:38.807911 (+2029720us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:40.973042 (+2165131us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:58:40.973908 (+ 866us) catalog_manager.cc:3725] Apply alter schema
0501 13:58:40.974747 (+ 839us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":280900,"Hive Metastore.run_cpu_time_us":1928,"Hive Metastore.run_wall_time_us":589725}
I20260501 13:58:40.978451 4061 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_f3eecd105941437fbf670fab2685bcf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:58:40.999302 4059 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9074c6e0ff374999bcf7953f17de8b4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:58:41.011009 4433 tablet_bootstrap.cc:492] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:41.011454 4433 ts_tablet_manager.cc:1403] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.097s user 0.012s sys 0.000s
I20260501 13:58:41.013846 4433 raft_consensus.cc:359] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.019564 4433 raft_consensus.cc:385] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:41.019870 4433 raft_consensus.cc:740] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:41.020486 4433 consensus_queue.cc:260] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.020918 4548 tablet.cc:1724] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:41.029390 4338 tablet_bootstrap.cc:492] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:41.034832 4433 ts_tablet_manager.cc:1434] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260501 13:58:41.034925 4543 tablet.cc:1724] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:41.039286 4338 tablet_bootstrap.cc:654] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:41.043292 4542 tablet.cc:1724] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 13:58:41.047506 4052 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:41.054307 4064 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet 1505e4bc749b48a88008d732881ff148
I20260501 13:58:41.066860 4524 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:41.067423 4524 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:41.070605 4524 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:41.072286 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 13:58:41.072917 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:41.073266 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:41.073928 4524 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:58:41.076567 4560 log.cc:927] Time spent T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260501 13:58:41.092559 4064 catalog_manager.cc:6741] ReplaceTablet: tablet 1505e4bc749b48a88008d732881ff148 deleted and replaced by tablet 0c215522a3a44450922f03cf2cb7cac3
I20260501 13:58:41.096705 3651 tablet_service.cc:1558] Processing DeleteTablet for tablet 1505e4bc749b48a88008d732881ff148 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c215522a3a44450922f03cf2cb7cac3) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:41.097566 4358 tablet_replica.cc:333] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:41.098217 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 1505e4bc749b48a88008d732881ff148 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c215522a3a44450922f03cf2cb7cac3) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:41.098366 4358 raft_consensus.cc:2243] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:41.098910 4549 tablet_replica.cc:333] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:41.099244 4358 raft_consensus.cc:2272] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:41.099426 4338 tablet_bootstrap.cc:492] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:41.099709 4549 raft_consensus.cc:2243] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:41.099817 4338 ts_tablet_manager.cc:1403] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.003s
I20260501 13:58:41.100217 4549 raft_consensus.cc:2272] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:41.101985 4358 ts_tablet_manager.cc:1916] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:41.102401 4549 ts_tablet_manager.cc:1916] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:41.102293 4338 raft_consensus.cc:359] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.102936 4338 raft_consensus.cc:385] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:41.103173 4338 raft_consensus.cc:740] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:41.103880 4338 consensus_queue.cc:260] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
W20260501 13:58:41.120898 4037 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.189:44255, user_credentials={real_user=slave}} blocked reactor thread for 45733.3us
I20260501 13:58:41.122586 3491 tablet_service.cc:1558] Processing DeleteTablet for tablet 1505e4bc749b48a88008d732881ff148 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c215522a3a44450922f03cf2cb7cac3) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:41.135190 4358 ts_tablet_manager.cc:1929] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:41.135653 4358 log.cc:1199] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/1505e4bc749b48a88008d732881ff148
I20260501 13:58:41.136574 4358 ts_tablet_manager.cc:1950] T 1505e4bc749b48a88008d732881ff148 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:41.156273 4338 ts_tablet_manager.cc:1434] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.056s user 0.007s sys 0.000s
I20260501 13:58:41.162770 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 1505e4bc749b48a88008d732881ff148 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]) successfully deleted
I20260501 13:58:41.162397 4063 catalog_manager.cc:5671] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:41.164534 4549 ts_tablet_manager.cc:1929] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:41.174978 4050 master_service.cc:954] ReplaceTablet: received request to replace tablet 6e76ac9fdeed442b94510860cd439b24 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:41.179785 4549 log.cc:1199] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/1505e4bc749b48a88008d732881ff148
I20260501 13:58:41.180449 3651 tablet_service.cc:1511] Processing CreateTablet for tablet 5be1b437dd2c4e8d99cd236038040d86 (DEFAULT_TABLE table=default.table_82ebd50001e848a8b155dead3f78ea41 [id=18405c74f9e14a2980c453b17acd0ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:41.181456 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be1b437dd2c4e8d99cd236038040d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:41.181474 4549 ts_tablet_manager.cc:1950] T 1505e4bc749b48a88008d732881ff148 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:41.184559 4064 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_0da08f161093429e83b826825c3a9727" } modify_external_catalogs: true
I20260501 13:58:41.185190 4064 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_0da08f161093429e83b826825c3a9727" } modify_external_catalogs: true
I20260501 13:58:41.186405 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 5be1b437dd2c4e8d99cd236038040d86 (DEFAULT_TABLE table=default.table_82ebd50001e848a8b155dead3f78ea41 [id=18405c74f9e14a2980c453b17acd0ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:41.187443 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be1b437dd2c4e8d99cd236038040d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:41.191396 3491 tablet_service.cc:1511] Processing CreateTablet for tablet 5be1b437dd2c4e8d99cd236038040d86 (DEFAULT_TABLE table=default.table_82ebd50001e848a8b155dead3f78ea41 [id=18405c74f9e14a2980c453b17acd0ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:41.192057 3800 tablet_service.cc:1511] Processing CreateTablet for tablet 322b71262eb44077bbe0cfcebc108882 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:41.192392 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be1b437dd2c4e8d99cd236038040d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:41.193100 3800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322b71262eb44077bbe0cfcebc108882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:41.200019 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 322b71262eb44077bbe0cfcebc108882 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:41.201122 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322b71262eb44077bbe0cfcebc108882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:41.190114 3492 tablet_service.cc:1511] Processing CreateTablet for tablet 322b71262eb44077bbe0cfcebc108882 (DEFAULT_TABLE table=default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:41.204398 3492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322b71262eb44077bbe0cfcebc108882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:41.204258 4036 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 1505e4bc749b48a88008d732881ff148 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]) successfully deleted
I20260501 13:58:41.206398 4498 raft_consensus.cc:493] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:41.206885 4498 raft_consensus.cc:515] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.209209 4498 leader_election.cc:290] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:41.210896 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70c23102f4004fcc83ce8e492b4f1d54" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:41.211539 3672 raft_consensus.cc:2468] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:41.212400 4341 tablet_bootstrap.cc:492] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:41.212612 3756 leader_election.cc:304] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:41.212970 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70c23102f4004fcc83ce8e492b4f1d54" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:41.213346 4498 raft_consensus.cc:2804] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:41.213578 3513 raft_consensus.cc:2468] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:41.213670 4498 raft_consensus.cc:493] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:41.213953 4498 raft_consensus.cc:3060] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:41.218246 4341 tablet_bootstrap.cc:654] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:41.233702 4498 raft_consensus.cc:515] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.238343 4341 tablet_bootstrap.cc:492] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:41.238955 4341 ts_tablet_manager.cc:1403] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.002s
I20260501 13:58:41.241634 4341 raft_consensus.cc:359] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.242982 4063 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_124cdf87baa148fbbc6e5f06d1efcda1" } new_table_name: "default.Table_83691904199e4bcaa0dd9732f655dec6" modify_external_catalogs: true
I20260501 13:58:41.243170 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet e12f37ec001949609d92e9118cbd9581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:40 UTC) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:41.243826 4358 tablet_replica.cc:333] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:41.244534 4358 raft_consensus.cc:2243] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:41.245010 4358 raft_consensus.cc:2272] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:41.249425 4358 ts_tablet_manager.cc:1916] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:41.252180 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70c23102f4004fcc83ce8e492b4f1d54" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:41.252710 3513 raft_consensus.cc:3060] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:41.251080 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70c23102f4004fcc83ce8e492b4f1d54" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:41.253206 3672 raft_consensus.cc:3060] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:41.253510 4341 raft_consensus.cc:385] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:41.253799 4341 raft_consensus.cc:740] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:41.257777 4341 consensus_queue.cc:260] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.260514 3513 raft_consensus.cc:2468] T 70c23102f4004fcc83ce8e492b4f1d54 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:41.263679 3672 raft_consensus.cc:2468] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:41.260993 4498 leader_election.cc:290] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:41.262066 3757 leader_election.cc:304] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:41.272037 4591 raft_consensus.cc:2804] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:41.272531 4591 raft_consensus.cc:697] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:41.273200 4561 raft_consensus.cc:493] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:41.273343 4591 consensus_queue.cc:237] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.273633 4561 raft_consensus.cc:515] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.276854 4561 leader_election.cc:290] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:41.278409 4358 ts_tablet_manager.cc:1929] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260501 13:58:41.280226 3757 leader_election.cc:343] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Not found: Tablet not found: 5be1b437dd2c4e8d99cd236038040d86
I20260501 13:58:41.282549 4341 ts_tablet_manager.cc:1434] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.043s user 0.005s sys 0.001s
I20260501 13:58:41.284487 4358 log.cc:1199] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/e12f37ec001949609d92e9118cbd9581
I20260501 13:58:41.285286 4358 ts_tablet_manager.cc:1950] T e12f37ec001949609d92e9118cbd9581 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:41.289317 4050 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262: Sending DeleteTablet for 3 replicas of tablet 6e76ac9fdeed442b94510860cd439b24
I20260501 13:58:41.290714 4050 catalog_manager.cc:6741] ReplaceTablet: tablet 6e76ac9fdeed442b94510860cd439b24 deleted and replaced by tablet 6087f52d19984dc2a83b6f923685610a
I20260501 13:58:41.292127 3802 tablet_service.cc:1558] Processing DeleteTablet for tablet 6e76ac9fdeed442b94510860cd439b24 with delete_type TABLET_DATA_DELETED (replaced by tablet 6087f52d19984dc2a83b6f923685610a) from {username='slave'} at 127.0.0.1:47972
I20260501 13:58:41.292841 4549 tablet_replica.cc:333] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:41.293669 4549 raft_consensus.cc:2243] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:41.294175 4549 raft_consensus.cc:2272] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:41.295004 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet 6e76ac9fdeed442b94510860cd439b24 with delete_type TABLET_DATA_DELETED (replaced by tablet 6087f52d19984dc2a83b6f923685610a) from {username='slave'} at 127.0.0.1:52150
I20260501 13:58:41.296043 4549 ts_tablet_manager.cc:1916] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:41.297214 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet 6e76ac9fdeed442b94510860cd439b24 with delete_type TABLET_DATA_DELETED (replaced by tablet 6087f52d19984dc2a83b6f923685610a) from {username='slave'} at 127.0.0.1:48686
I20260501 13:58:41.298209 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet e12f37ec001949609d92e9118cbd9581 (table default.table_e7ee55876f2f41cfa624a58f23f9d415 [id=8d51c67732064782bd5d53da7178ba87]) successfully deleted
I20260501 13:58:41.303334 4341 tablet_bootstrap.cc:492] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:41.305757 4358 tablet_replica.cc:333] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:41.306710 4358 raft_consensus.cc:2243] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:41.307219 4358 raft_consensus.cc:2272] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:41.308954 4358 ts_tablet_manager.cc:1916] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:41.311813 4524 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:41.312352 4524 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:41.300243 4052 catalog_manager.cc:5671] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9154033cea374a10a5225c53b8bcf122 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "9154033cea374a10a5225c53b8bcf122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:41.324669 4524 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:41.333838 4357 ts_tablet_manager.cc:1929] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:41.339447 4341 tablet_bootstrap.cc:654] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:41.353634 4549 ts_tablet_manager.cc:1929] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:41.362668 4358 ts_tablet_manager.cc:1929] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260501 13:58:41.363020 4358 log.cc:1199] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/6e76ac9fdeed442b94510860cd439b24
I20260501 13:58:41.363628 4358 ts_tablet_manager.cc:1950] T 6e76ac9fdeed442b94510860cd439b24 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:41.366106 4036 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 6e76ac9fdeed442b94510860cd439b24 (table default.table_124cdf87baa148fbbc6e5f06d1efcda1 [id=4f022a0cee6540b5959fd48b029df1af]) successfully deleted
I20260501 13:58:41.370014 4549 log.cc:1199] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/6e76ac9fdeed442b94510860cd439b24
I20260501 13:58:41.372963 4549 ts_tablet_manager.cc:1950] T 6e76ac9fdeed442b94510860cd439b24 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:41.380465 4341 tablet_bootstrap.cc:492] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:41.386051 4341 ts_tablet_manager.cc:1403] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.083s user 0.009s sys 0.004s
I20260501 13:58:41.388345 4052 master_service.cc:954] ReplaceTablet: received request to replace tablet 6e76ac9fdeed442b94510860cd439b24 from {username='slave'} at 127.0.0.1:35438
I20260501 13:58:41.397358 4341 raft_consensus.cc:359] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:41.409536 4341 raft_consensus.cc:385] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:41.409938 4341 raft_consensus.cc:740] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
W20260501 13:58:41.410600 4037 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6b0f8911dd5d431d9d8d3124a4456262 -> Peer 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255): Couldn't send request to peer 15dd70d7c5494c3c911f0a780dc9a2af. Status: Network error: Client connection negotiation failed: client connection to 127.1.16.189:44255: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260501 13:58:41.412580 4341 consensus_queue.cc:260] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:41.416723 4338 tablet_bootstrap.cc:492] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
W20260501 13:58:41.417192 3756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.130:45857, user_credentials={real_user=slave}} blocked reactor thread for 144302us
I20260501 13:58:41.424218 4341 ts_tablet_manager.cc:1434] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.029s user 0.004s sys 0.001s
I20260501 13:58:41.427974 4338 tablet_bootstrap.cc:654] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
W20260501 13:58:41.439729 3756 leader_election.cc:343] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 5be1b437dd2c4e8d99cd236038040d86
I20260501 13:58:41.440405 3756 leader_election.cc:304] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2; no voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122
I20260501 13:58:41.442876 4591 raft_consensus.cc:2749] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 13:58:41.392961 4433 tablet_bootstrap.cc:492] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
W20260501 13:58:41.407721 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:58:41.455562 3448 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.131:38685, user_credentials={real_user=slave}} blocked reactor thread for 47942.2us
W20260501 13:58:41.456421 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:58:41.456766 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:41.465476 4433 tablet_bootstrap.cc:654] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:41.469192 4524 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:41.471817 4056 catalog_manager.cc:3502] renamed table default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9] in HMS: new name default.table_3240c40926184fd7b92c22b821de47e9
I20260501 13:58:41.473580 4357 log.cc:1199] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/65516a8e9190491587780880f6e1bbf9
I20260501 13:58:41.474535 4357 ts_tablet_manager.cc:1950] T 65516a8e9190491587780880f6e1bbf9 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:41.477152 4037 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 65516a8e9190491587780880f6e1bbf9 (table default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]) successfully deleted
I20260501 13:58:41.482694 4338 tablet_bootstrap.cc:492] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:41.483209 4338 ts_tablet_manager.cc:1403] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.000s
I20260501 13:58:41.500460 4187 catalog_manager.cc:2772] Deleting table default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2] in response to Hive Metastore notification log event 30
W20260501 13:58:41.506799 3448 consensus_peers.cc:597] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e76ac9fdeed442b94510860cd439b24. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:58:41.517799 4433 tablet_bootstrap.cc:492] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:41.518354 4433 ts_tablet_manager.cc:1403] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.126s user 0.010s sys 0.000s
W20260501 13:58:41.524897 3448 consensus_peers.cc:597] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 -> Peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Couldn't send request to peer 5c239ccbdc554619a237d78e958ffbc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e76ac9fdeed442b94510860cd439b24. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:58:41.521775 4433 raft_consensus.cc:359] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:41.525815 4433 raft_consensus.cc:385] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:41.526134 4433 raft_consensus.cc:740] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:41.526975 4433 consensus_queue.cc:260] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 13:58:41.537806 4037 catalog_manager.cc:4969] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet e12f37ec001949609d92e9118cbd9581 because the tablet was not found. No further retry: Not found: Tablet not found: e12f37ec001949609d92e9118cbd9581
I20260501 13:58:41.538997 4357 tablet_replica.cc:333] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:41.540699 4357 raft_consensus.cc:2243] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:41.486377 4338 raft_consensus.cc:359] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:41.547672 4338 raft_consensus.cc:385] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:41.555681 4357 raft_consensus.cc:2272] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:41.556205 4338 raft_consensus.cc:740] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:41.556416 4620 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.16.189:44255 every 8 connection(s)
I20260501 13:58:41.560576 4309 rpc_server.cc:307] RPC server started. Bound to: 127.1.16.189:44255
I20260501 13:58:41.562935 4357 ts_tablet_manager.cc:1916] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:41.564441 4338 consensus_queue.cc:260] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:41.564893 4309 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/master-1/data/info.pb
I20260501 13:58:41.567631 1090 external_mini_cluster.cc:1428] Started /tmp/dist-test-task2LI5eU/build/tsan/bin/kudu as pid 4309
I20260501 13:58:41.572084 4059 catalog_manager.cc:2174] created HMS catalog entry for table [id=d12f8a5ea4904fad99a4749334d47412]
I20260501 13:58:41.595573 4622 sys_catalog.cc:263] Verifying existing consensus state
I20260501 13:58:41.559712 4433 ts_tablet_manager.cc:1434] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20260501 13:58:41.605252 4622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Bootstrap starting.
I20260501 13:58:41.612473 4433 tablet_bootstrap.cc:492] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:41.612241 4586 consensus_queue.cc:1048] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:41.621309 4338 ts_tablet_manager.cc:1434] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.137s user 0.006s sys 0.000s
I20260501 13:58:41.643139 4338 tablet_bootstrap.cc:492] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:41.659403 3562 heartbeater.cc:344] Connected to a master server at 127.1.16.189:44255
I20260501 13:58:41.693118 4433 tablet_bootstrap.cc:654] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:41.703595 4586 consensus_queue.cc:1048] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:41.721046 4357 ts_tablet_manager.cc:1929] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 13:58:41.710355 3720 heartbeater.cc:344] Connected to a master server at 127.1.16.189:44255
I20260501 13:58:41.711107 4487 consensus_queue.cc:1048] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260501 13:58:41.746870 3868 heartbeater.cc:344] Connected to a master server at 127.1.16.189:44255
I20260501 13:58:41.746682 4357 log.cc:1199] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/1505e4bc749b48a88008d732881ff148
I20260501 13:58:41.748206 4357 ts_tablet_manager.cc:1950] T 1505e4bc749b48a88008d732881ff148 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:41.749270 4357 tablet_replica.cc:333] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:41.749974 4357 raft_consensus.cc:2243] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:41.750844 4357 pending_rounds.cc:70] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: Trying to abort 1 pending ops.
I20260501 13:58:41.752413 4338 tablet_bootstrap.cc:654] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:41.751058 4357 pending_rounds.cc:77] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7281229498138947584 op_type: NO_OP noop_request { }
I20260501 13:58:41.766762 4357 raft_consensus.cc:2889] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 13:58:41.767151 4357 raft_consensus.cc:2272] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:41.769160 4357 ts_tablet_manager.cc:1916] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:41.776492 4433 tablet_bootstrap.cc:492] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:41.777328 4433 ts_tablet_manager.cc:1403] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.165s user 0.010s sys 0.002s
I20260501 13:58:41.781901 4433 raft_consensus.cc:359] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.804618 4433 raft_consensus.cc:385] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:41.805771 4433 raft_consensus.cc:740] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:41.807265 4433 consensus_queue.cc:260] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.815788 4433 ts_tablet_manager.cc:1434] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20260501 13:58:41.827442 4357 ts_tablet_manager.cc:1929] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:41.827840 4357 log.cc:1199] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/6e76ac9fdeed442b94510860cd439b24
I20260501 13:58:41.828545 4357 ts_tablet_manager.cc:1950] T 6e76ac9fdeed442b94510860cd439b24 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:41.842324 4622 log.cc:826] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Log is configured to *not* fsync() on all Append() calls
I20260501 13:58:41.864845 4569 consensus_queue.cc:1048] T eb5de60e4d1041f29ef1ecb2d0181a26 P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:41.879424 4595 raft_consensus.cc:493] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:41.879930 4595 raft_consensus.cc:515] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:41.882730 4595 leader_election.cc:290] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:41.882790 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322b71262eb44077bbe0cfcebc108882" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:41.883415 3672 raft_consensus.cc:2468] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:41.883873 3868 heartbeater.cc:344] Connected to a master server at 127.1.16.189:44255
I20260501 13:58:41.884883 3448 leader_election.cc:304] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:41.886317 4595 raft_consensus.cc:2804] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:41.886684 4595 raft_consensus.cc:493] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:41.886478 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322b71262eb44077bbe0cfcebc108882" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:41.887022 4595 raft_consensus.cc:3060] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:41.887179 3822 raft_consensus.cc:2468] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:41.931318 4639 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:41.936409 4338 tablet_bootstrap.cc:492] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:41.936877 4338 ts_tablet_manager.cc:1403] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.294s user 0.016s sys 0.008s
I20260501 13:58:41.939549 4338 raft_consensus.cc:359] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.936957 4639 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:41.940164 4338 raft_consensus.cc:385] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:41.940436 4338 raft_consensus.cc:740] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:41.941100 4338 consensus_queue.cc:260] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:41.946293 4338 ts_tablet_manager.cc:1434] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20260501 13:58:41.952987 4639 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:41.985401 3720 heartbeater.cc:344] Connected to a master server at 127.1.16.189:44255
I20260501 13:58:41.995321 4595 raft_consensus.cc:515] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:42.019245 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322b71262eb44077bbe0cfcebc108882" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:42.019821 3672 raft_consensus.cc:3060] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:42.023720 3820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "322b71262eb44077bbe0cfcebc108882" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:42.024326 3820 raft_consensus.cc:3060] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:42.022957 4595 leader_election.cc:290] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:42.037321 3820 raft_consensus.cc:2468] T 322b71262eb44077bbe0cfcebc108882 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:42.038736 3448 leader_election.cc:304] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:42.039666 4595 raft_consensus.cc:2804] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:42.039973 3672 raft_consensus.cc:2468] T 322b71262eb44077bbe0cfcebc108882 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:42.042900 4595 raft_consensus.cc:697] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:42.052261 4595 consensus_queue.cc:237] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:42.105733 4595 raft_consensus.cc:493] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:42.106187 4595 raft_consensus.cc:515] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:42.108091 4595 leader_election.cc:290] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:42.110376 3820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be1b437dd2c4e8d99cd236038040d86" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:42.111022 3820 raft_consensus.cc:2468] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:42.112018 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be1b437dd2c4e8d99cd236038040d86" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:42.112201 3448 leader_election.cc:304] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:42.112620 3672 raft_consensus.cc:2468] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:58:42.113701 4595 raft_consensus.cc:2804] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:42.114029 4595 raft_consensus.cc:493] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:42.114358 4595 raft_consensus.cc:3060] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:42.121152 4595 raft_consensus.cc:515] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:42.123095 4595 leader_election.cc:290] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:42.124158 3820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be1b437dd2c4e8d99cd236038040d86" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:42.124713 3820 raft_consensus.cc:3060] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:42.124804 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be1b437dd2c4e8d99cd236038040d86" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:42.125313 3672 raft_consensus.cc:3060] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:42.140717 3820 raft_consensus.cc:2468] T 5be1b437dd2c4e8d99cd236038040d86 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:42.142057 3448 leader_election.cc:304] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:42.154577 3672 raft_consensus.cc:2468] T 5be1b437dd2c4e8d99cd236038040d86 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:58:42.156227 4595 raft_consensus.cc:2804] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:42.159508 4595 raft_consensus.cc:697] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:58:42.160439 4595 consensus_queue.cc:237] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:42.167399 4590 consensus_queue.cc:1048] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:58:42.252650 4590 consensus_queue.cc:1048] T 70c23102f4004fcc83ce8e492b4f1d54 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260501 13:58:42.527666 3140 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.1.16.189:44255 timed out after 0.583s (SENT)
W20260501 13:58:42.740255 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 273ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:42.896719 4595 consensus_queue.cc:1048] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:42.986922 3562 heartbeater.cc:344] Connected to a master server at 127.1.16.189:44255
I20260501 13:58:43.006829 4586 consensus_queue.cc:1048] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:43.016770 4622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Bootstrap replayed 1/1 log segments. Stats: ops{read=111 overwritten=0 applied=111 ignored=0} inserts{seen=61 ignored=0} mutations{seen=134 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 13:58:43.017784 4622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: Bootstrap complete.
I20260501 13:58:43.025712 4658 consensus_queue.cc:1048] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:43.113303 4586 consensus_queue.cc:1048] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:58:43.123643 3140 leader_election.cc:336] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255): Timed out: RequestConsensusVote RPC to 127.1.16.189:44255 timed out after 0.583s (SENT)
I20260501 13:58:43.120692 4622 raft_consensus.cc:359] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
W20260501 13:58:43.124192 3139 leader_election.cc:336] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253): Timed out: RequestConsensusVote RPC to 127.1.16.188:38253 timed out after 0.583s (SENT)
I20260501 13:58:43.124472 4622 raft_consensus.cc:740] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15dd70d7c5494c3c911f0a780dc9a2af, State: Initialized, Role: FOLLOWER
I20260501 13:58:43.124569 3139 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016; no voters: 15dd70d7c5494c3c911f0a780dc9a2af, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:43.125366 4622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 111, Last appended: 5.111, Last appended by leader: 111, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:43.159801 4666 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260501 13:58:43.169284 4622 sys_catalog.cc:565] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: configured and running, proceeding with master startup.
I20260501 13:58:43.169430 4587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 7 candidate_status { last_received { term: 6 index: 203 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:43.170372 4587 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a1d2cd221924677a3ef81b081e5e016 in term 5.
W20260501 13:58:43.170948 4587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49094 (request call id 0) took 1157 ms (client timeout 583 ms). Trace:
W20260501 13:58:43.171303 4587 rpcz_store.cc:269] 0501 13:58:42.013387 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:42.013516 (+ 129us) service_pool.cc:224] Handling call
0501 13:58:43.170871 (+1157355us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260501 13:58:43.175470 4584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 7 candidate_status { last_received { term: 6 index: 203 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:43.176128 4584 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a1d2cd221924677a3ef81b081e5e016 in term 5.
W20260501 13:58:43.176628 4584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49094 (request call id 1) took 637 ms (client timeout 583 ms). Trace:
W20260501 13:58:43.176915 4584 rpcz_store.cc:269] 0501 13:58:42.539496 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:42.539607 (+ 111us) service_pool.cc:224] Handling call
0501 13:58:43.176569 (+636962us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 13:58:43.181581 4038 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b9f3b4229594e13959fec0aa8281c03 consensus state has no leader
I20260501 13:58:43.191282 4667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:43.204087 4667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:43.340189 4575 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" instance_seqno: 1777643902876599) as {username='slave'} at 127.1.16.131:44075; Asking this server to re-register.
I20260501 13:58:43.345211 3868 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:43.359190 4575 ts_manager.cc:194] Registered new tserver with Master: 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:43.398504 4691 catalog_manager.cc:1269] Loaded cluster ID: 4281e294e62c41c781e2041e1b24de7c
I20260501 13:58:43.410537 4691 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: loading cluster ID for follower catalog manager: success
I20260501 13:58:43.441591 4691 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: acquiring CA information for follower catalog manager: success
I20260501 13:58:43.455155 4691 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260501 13:58:43.558809 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 273ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:43.596823 4575 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9154033cea374a10a5225c53b8bcf122" instance_seqno: 1777643898269658) as {username='slave'} at 127.1.16.129:47691; Asking this server to re-register.
I20260501 13:58:43.599020 3562 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:43.605360 4575 ts_manager.cc:194] Registered new tserver with Master: 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:43.678068 4702 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:43.678694 4702 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:43.686113 4584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 7 candidate_status { last_received { term: 6 index: 203 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" is_pre_election: true
I20260501 13:58:43.686784 4584 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a1d2cd221924677a3ef81b081e5e016 in term 5.
I20260501 13:58:43.688040 4575 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" instance_seqno: 1777643900635713) as {username='slave'} at 127.1.16.130:55895; Asking this server to re-register.
I20260501 13:58:43.688935 4702 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:43.688908 3140 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016, 15dd70d7c5494c3c911f0a780dc9a2af; no voters:
I20260501 13:58:43.690927 4702 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260501 13:58:43.691783 4702 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6b0f8911dd5d431d9d8d3124a4456262)
I20260501 13:58:43.692198 4702 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 6 FOLLOWER]: Advancing to term 7
I20260501 13:58:43.692448 3720 heartbeater.cc:461] Registering TS with master...
I20260501 13:58:43.699534 4575 ts_manager.cc:194] Registered new tserver with Master: 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857)
I20260501 13:58:43.701301 4702 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:43.705751 4584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a1d2cd221924677a3ef81b081e5e016" candidate_term: 7 candidate_status { last_received { term: 6 index: 203 } } ignore_live_leader: false dest_uuid: "15dd70d7c5494c3c911f0a780dc9a2af"
I20260501 13:58:43.706331 4584 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 5 FOLLOWER]: Advancing to term 7
I20260501 13:58:43.717365 4584 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a1d2cd221924677a3ef81b081e5e016 in term 7.
I20260501 13:58:43.718397 4702 leader_election.cc:290] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 7 election: Requested vote from peers 15dd70d7c5494c3c911f0a780dc9a2af (127.1.16.189:44255), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:43.719821 3140 leader_election.cc:304] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a1d2cd221924677a3ef81b081e5e016, 15dd70d7c5494c3c911f0a780dc9a2af; no voters:
I20260501 13:58:43.720657 4702 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 7 FOLLOWER]: Leader election won for term 7
I20260501 13:58:43.722751 4702 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [term 7 LEADER]: Becoming Leader. State: Replica: 0a1d2cd221924677a3ef81b081e5e016, State: Running, Role: LEADER
I20260501 13:58:43.724571 4702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 203, Committed index: 203, Last appended: 6.203, Last appended by leader: 203, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:43.730358 4703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a1d2cd221924677a3ef81b081e5e016. Latest consensus state: current_term: 7 leader_uuid: "0a1d2cd221924677a3ef81b081e5e016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:43.731158 4703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: This master's current role is: LEADER
I20260501 13:58:43.747591 4705 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 13:58:43.778195 4705 catalog_manager.cc:679] Loaded metadata for table default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942]
I20260501 13:58:43.792217 4705 catalog_manager.cc:679] Loaded metadata for table default.table_82ebd50001e848a8b155dead3f78ea41 [id=18405c74f9e14a2980c453b17acd0ac9]
I20260501 13:58:43.794114 4705 catalog_manager.cc:679] Loaded metadata for table default.table_04a1387d0f0343998d94c1bb5bbff699 [id=18ac75baaaf7485da21228e06a780ad2]
I20260501 13:58:43.795944 4705 catalog_manager.cc:679] Loaded metadata for table default.table_124cdf87baa148fbbc6e5f06d1efcda1 [id=4f022a0cee6540b5959fd48b029df1af]
I20260501 13:58:43.798385 4705 catalog_manager.cc:679] Loaded metadata for table default.table_c0462a887c864245890f76dea029e11b [id=52fc296b289947f1be9e9f74d320675c]
I20260501 13:58:43.802274 4705 catalog_manager.cc:679] Loaded metadata for table default.table_f52f85ad11fb4023adead2ab9c5eb694 [id=53fa808a27054cdf834732393dc08838]
I20260501 13:58:43.812269 4705 catalog_manager.cc:679] Loaded metadata for table default.table_0da08f161093429e83b826825c3a9727 [id=5b3f55db8abb4fa7a9b3abf286620942]
I20260501 13:58:43.814057 4705 catalog_manager.cc:679] Loaded metadata for table default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9]
I20260501 13:58:43.815829 4705 catalog_manager.cc:679] Loaded metadata for table default.table_59dc65b11ade4ee3acb91116f7a60585 [id=9385850ae45c48ca955d499dd499a90b]
I20260501 13:58:43.817792 4705 catalog_manager.cc:679] Loaded metadata for table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]
I20260501 13:58:43.820809 4705 catalog_manager.cc:679] Loaded metadata for table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef]
I20260501 13:58:43.828068 4705 catalog_manager.cc:679] Loaded metadata for table default.table_8be6ed54e88344f7b4be783af8fa48da [id=eac59b190539443d88199a65abe479b9]
I20260501 13:58:43.829879 4705 catalog_manager.cc:679] Loaded metadata for table default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]
I20260501 13:58:43.860374 4705 tablet_loader.cc:96] loaded metadata for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab])
I20260501 13:58:43.861986 4705 tablet_loader.cc:96] loaded metadata for tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 (table default.table_0da08f161093429e83b826825c3a9727 [id=5b3f55db8abb4fa7a9b3abf286620942])
I20260501 13:58:43.863492 4705 tablet_loader.cc:96] loaded metadata for tablet 322b71262eb44077bbe0cfcebc108882 (table default.table_b4b1da6bbecc440286963f934507777c [id=095cec654bd6468da1aa74b795dc7942])
I20260501 13:58:43.867602 4705 tablet_loader.cc:96] loaded metadata for tablet 41ac3120d65440829fe929b76cc41d43 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef])
I20260501 13:58:43.868965 4705 tablet_loader.cc:96] loaded metadata for tablet 533eb8337c414aea98ea95d7bf507b1e (table default.table_f52f85ad11fb4023adead2ab9c5eb694 [id=53fa808a27054cdf834732393dc08838])
I20260501 13:58:43.871584 4705 tablet_loader.cc:96] loaded metadata for tablet 545ced72d1cd47faa827d7b00df372b7 (table default.table_04a1387d0f0343998d94c1bb5bbff699 [id=18ac75baaaf7485da21228e06a780ad2])
I20260501 13:58:43.875597 4705 tablet_loader.cc:96] loaded metadata for tablet 5be1b437dd2c4e8d99cd236038040d86 (table default.table_82ebd50001e848a8b155dead3f78ea41 [id=18405c74f9e14a2980c453b17acd0ac9])
I20260501 13:58:43.876809 4705 tablet_loader.cc:96] loaded metadata for tablet 6087f52d19984dc2a83b6f923685610a (table default.table_124cdf87baa148fbbc6e5f06d1efcda1 [id=4f022a0cee6540b5959fd48b029df1af])
I20260501 13:58:43.884476 4705 tablet_loader.cc:96] loaded metadata for tablet 70c23102f4004fcc83ce8e492b4f1d54 (table default.table_8be6ed54e88344f7b4be783af8fa48da [id=eac59b190539443d88199a65abe479b9])
I20260501 13:58:43.885876 4705 tablet_loader.cc:96] loaded metadata for tablet 776eacf49dbd4c2f9a8dfa1ae2181c96 (table default.table_f701ba5ffc3845bf8cb5e55102e60688 [id=748e365ce7f04451b1157e00a8813bb9])
I20260501 13:58:43.887305 4705 tablet_loader.cc:96] loaded metadata for tablet addbcaae1b464378aabe7bc7f4bcc7fa (table default.table_c0462a887c864245890f76dea029e11b [id=52fc296b289947f1be9e9f74d320675c])
I20260501 13:58:43.888664 4705 tablet_loader.cc:96] loaded metadata for tablet df6234730e704c77a3a85a407f0e1cff (table default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2])
I20260501 13:58:43.891592 4705 tablet_loader.cc:96] loaded metadata for tablet eb5de60e4d1041f29ef1ecb2d0181a26 (table default.table_59dc65b11ade4ee3acb91116f7a60585 [id=9385850ae45c48ca955d499dd499a90b])
I20260501 13:58:43.903728 4705 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 13:58:43.931614 4705 catalog_manager.cc:1269] Loaded cluster ID: 4281e294e62c41c781e2041e1b24de7c
I20260501 13:58:43.932016 4705 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 13:58:43.956427 4705 catalog_manager.cc:1514] Loading token signing keys...
I20260501 13:58:43.972443 4705 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: Loaded TSK: 0
I20260501 13:58:43.983688 4705 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 13:58:43.995661 4705 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260501 13:58:44.013861 4705 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: 29
I20260501 13:58:44.088905 3417 catalog_manager.cc:2772] Deleting table default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2] in response to Hive Metastore notification log event 30
I20260501 13:58:44.119758 4584 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Refusing update from remote peer 0a1d2cd221924677a3ef81b081e5e016: Log matching property violated. Preceding OpId in replica: term: 5 index: 111. Preceding OpId from leader: term: 7 index: 205. (index mismatch)
I20260501 13:58:44.121562 4703 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 111, Time since last communication: 0.000s
W20260501 13:58:44.122026 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 2222ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:44.499967 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 6: this message will repeat every 5th retry.
I20260501 13:58:44.550089 4707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a1d2cd221924677a3ef81b081e5e016. Latest consensus state: current_term: 7 leader_uuid: "0a1d2cd221924677a3ef81b081e5e016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:44.551131 4707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:44.561331 4711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "0a1d2cd221924677a3ef81b081e5e016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:44.564479 4711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:44.557935 4707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "0a1d2cd221924677a3ef81b081e5e016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:44.571604 4707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [sys.catalog]: This master's current role is: FOLLOWER
I20260501 13:58:44.567159 4715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "0a1d2cd221924677a3ef81b081e5e016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } } }
I20260501 13:58:44.573084 4715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [sys.catalog]: This master's current role is: LEADER
I20260501 13:58:44.582248 3417 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: Sending DeleteTablet for 3 replicas of tablet df6234730e704c77a3a85a407f0e1cff
I20260501 13:58:44.592821 3154 catalog_manager.cc:5671] T 322b71262eb44077bbe0cfcebc108882 P 9154033cea374a10a5225c53b8bcf122 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9154033cea374a10a5225c53b8bcf122 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "9154033cea374a10a5225c53b8bcf122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } }
W20260501 13:58:44.602936 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 2222ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:44.596520 3154 catalog_manager.cc:5671] T 5be1b437dd2c4e8d99cd236038040d86 P 9154033cea374a10a5225c53b8bcf122 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9154033cea374a10a5225c53b8bcf122 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "9154033cea374a10a5225c53b8bcf122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: HEALTHY } } }
I20260501 13:58:44.672238 3652 tablet_service.cc:1511] Processing CreateTablet for tablet 0c215522a3a44450922f03cf2cb7cac3 (DEFAULT_TABLE table=default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:44.673274 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c215522a3a44450922f03cf2cb7cac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:44.675161 3651 tablet_service.cc:1511] Processing CreateTablet for tablet 6087f52d19984dc2a83b6f923685610a (DEFAULT_TABLE table=default.table_124cdf87baa148fbbc6e5f06d1efcda1 [id=4f022a0cee6540b5959fd48b029df1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:44.676165 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6087f52d19984dc2a83b6f923685610a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:44.677313 3493 tablet_service.cc:1558] Processing DeleteTablet for tablet df6234730e704c77a3a85a407f0e1cff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:44 UTC) from {username='slave'} at 127.0.0.1:38966
I20260501 13:58:44.678805 3721 heartbeater.cc:499] Master 127.1.16.190:38561 was elected leader, sending a full tablet report...
I20260501 13:58:44.679992 3654 tablet_service.cc:1558] Processing DeleteTablet for tablet df6234730e704c77a3a85a407f0e1cff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:44 UTC) from {username='slave'} at 127.0.0.1:35134
I20260501 13:58:44.681425 3492 tablet_service.cc:1511] Processing CreateTablet for tablet 0c215522a3a44450922f03cf2cb7cac3 (DEFAULT_TABLE table=default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:44.682567 3492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c215522a3a44450922f03cf2cb7cac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:44.687288 3491 tablet_service.cc:1511] Processing CreateTablet for tablet 6087f52d19984dc2a83b6f923685610a (DEFAULT_TABLE table=default.table_124cdf87baa148fbbc6e5f06d1efcda1 [id=4f022a0cee6540b5959fd48b029df1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:44.688429 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6087f52d19984dc2a83b6f923685610a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:44.704968 4726 tablet_replica.cc:333] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 13:58:44.706939 4726 raft_consensus.cc:2243] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 13:58:44.709422 4726 raft_consensus.cc:2272] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:44.736521 4726 ts_tablet_manager.cc:1916] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:44.746749 3565 heartbeater.cc:499] Master 127.1.16.190:38561 was elected leader, sending a full tablet report...
W20260501 13:58:44.765717 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 13:58:44.765995 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
W20260501 13:58:44.766309 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:58:44.766629 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet ba69864093504babbb00fb6170774778
W20260501 13:58:44.766817 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
W20260501 13:58:44.766970 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:44.782413 3152 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 776eacf49dbd4c2f9a8dfa1ae2181c96 (table default.table_3240c40926184fd7b92c22b821de47e9 [id=748e365ce7f04451b1157e00a8813bb9]). Expected version 1 got 0
W20260501 13:58:44.784812 3609 consensus_peers.cc:597] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 -> Peer 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): Couldn't send request to peer 9154033cea374a10a5225c53b8bcf122. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260501 13:58:44.793213 3152 catalog_manager.cc:5671] T 70c23102f4004fcc83ce8e492b4f1d54 P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } }
I20260501 13:58:44.824043 4726 ts_tablet_manager.cc:1929] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:44.825711 4726 log.cc:1199] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/df6234730e704c77a3a85a407f0e1cff
I20260501 13:58:44.829656 4726 ts_tablet_manager.cc:1950] T df6234730e704c77a3a85a407f0e1cff P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 13:58:44.832944 3140 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet df6234730e704c77a3a85a407f0e1cff (table default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]) successfully deleted
I20260501 13:58:44.840070 4727 tablet_replica.cc:333] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 13:58:44.841185 4727 raft_consensus.cc:2243] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 13:58:44.842223 4727 raft_consensus.cc:2272] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 13:58:44.846056 3139 leader_election.cc:336] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253): Timed out: RequestConsensusVote RPC to 127.1.16.188:38253 timed out after 0.546s (SENT)
I20260501 13:58:44.849897 3869 heartbeater.cc:499] Master 127.1.16.190:38561 was elected leader, sending a full tablet report...
I20260501 13:58:44.858372 3801 tablet_service.cc:1558] Processing DeleteTablet for tablet df6234730e704c77a3a85a407f0e1cff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:44 UTC) from {username='slave'} at 127.0.0.1:46442
I20260501 13:58:44.858928 3802 tablet_service.cc:1511] Processing CreateTablet for tablet 0c215522a3a44450922f03cf2cb7cac3 (DEFAULT_TABLE table=default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:44.860365 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c215522a3a44450922f03cf2cb7cac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:58:44.863426 3800 tablet_service.cc:1511] Processing CreateTablet for tablet 6087f52d19984dc2a83b6f923685610a (DEFAULT_TABLE table=default.table_124cdf87baa148fbbc6e5f06d1efcda1 [id=4f022a0cee6540b5959fd48b029df1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:58:44.864535 3800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6087f52d19984dc2a83b6f923685610a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260501 13:58:44.870934 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 80a2f6fab87a4ad980c7816f34f41aff
W20260501 13:58:44.871649 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:44.879416 4733 tablet_replica.cc:333] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 13:58:44.888588 4659 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:44.889081 4659 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:44.891139 4659 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:58:44.890020 4733 raft_consensus.cc:2243] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260501 13:58:44.894110 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:58:44.894017 4733 raft_consensus.cc:2272] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 13:58:44.912730 4733 ts_tablet_manager.cc:1916] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 13:58:44.919310 4737 tablet.cc:1724] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 9154033cea374a10a5225c53b8bcf122: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:44.927170 3154 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 776eacf49dbd4c2f9a8dfa1ae2181c96 (table default.table_3240c40926184fd7b92c22b821de47e9 [id=748e365ce7f04451b1157e00a8813bb9]). Expected version 1 got 0
I20260501 13:58:44.923405 4738 tablet.cc:1724] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 5c239ccbdc554619a237d78e958ffbc2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:44.947960 3153 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 776eacf49dbd4c2f9a8dfa1ae2181c96 (table default.table_3240c40926184fd7b92c22b821de47e9 [id=748e365ce7f04451b1157e00a8813bb9]). Expected version 1 got 0
I20260501 13:58:44.997309 4727 ts_tablet_manager.cc:1916] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 13:58:45.001394 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:58:45.001807 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:45.002941 4659 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:45.001190 4739 tablet_bootstrap.cc:492] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:44.992339 3492 tablet_service.cc:1558] Processing DeleteTablet for tablet df6234730e704c77a3a85a407f0e1cff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 13:58:44 UTC) from {username='slave'} at 127.0.0.1:38966
W20260501 13:58:45.009436 3140 catalog_manager.cc:4969] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): delete failed for tablet df6234730e704c77a3a85a407f0e1cff because the tablet was not found. No further retry: Not found: Tablet not found: df6234730e704c77a3a85a407f0e1cff
I20260501 13:58:45.026170 4739 tablet_bootstrap.cc:654] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:45.030987 4727 ts_tablet_manager.cc:1929] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:45.031492 4727 log.cc:1199] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/df6234730e704c77a3a85a407f0e1cff
I20260501 13:58:45.032389 4727 ts_tablet_manager.cc:1950] T df6234730e704c77a3a85a407f0e1cff P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 13:58:45.035164 3139 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet df6234730e704c77a3a85a407f0e1cff (table default.table_aafd3ab3f9f44a74af41aae60cfe40fa [id=f6a8e3a12e48471fb8788a66cd890ea2]) successfully deleted
I20260501 13:58:45.056213 4742 tablet_bootstrap.cc:492] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:45.090070 4742 tablet_bootstrap.cc:654] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:45.047613 4743 tablet.cc:1724] T 776eacf49dbd4c2f9a8dfa1ae2181c96 P 6af0cedc7f3c47299b93e6e92d8e02d1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:45.041725 4740 tablet_bootstrap.cc:492] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:45.166827 4739 tablet_bootstrap.cc:492] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:45.167765 4739 ts_tablet_manager.cc:1403] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.171s user 0.032s sys 0.060s
I20260501 13:58:45.169476 4742 tablet_bootstrap.cc:492] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:45.171141 4742 ts_tablet_manager.cc:1403] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.005s
I20260501 13:58:45.171818 4739 raft_consensus.cc:359] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.177932 4739 raft_consensus.cc:385] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:45.178277 4739 raft_consensus.cc:740] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:45.179114 4739 consensus_queue.cc:260] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.185132 4739 ts_tablet_manager.cc:1434] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260501 13:58:45.186219 4739 tablet_bootstrap.cc:492] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:58:45.193019 4739 tablet_bootstrap.cc:654] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:45.202946 3152 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: default.table_3240c40926184fd7b92c22b821de47e9 [id=748e365ce7f04451b1157e00a8813bb9] alter complete (version 1)
I20260501 13:58:45.224669 4740 tablet_bootstrap.cc:654] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:45.231349 4739 tablet_bootstrap.cc:492] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:58:45.231763 4739 ts_tablet_manager.cc:1403] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.046s user 0.015s sys 0.005s
I20260501 13:58:45.233620 4742 raft_consensus.cc:359] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.234339 4739 raft_consensus.cc:359] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.242448 4739 raft_consensus.cc:385] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:45.242816 4739 raft_consensus.cc:740] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:58:45.243496 4739 consensus_queue.cc:260] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.241235 4742 raft_consensus.cc:385] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:45.245860 4739 ts_tablet_manager.cc:1434] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260501 13:58:45.244850 4742 raft_consensus.cc:740] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:45.266938 4733 ts_tablet_manager.cc:1929] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 13:58:45.267395 4733 log.cc:1199] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/df6234730e704c77a3a85a407f0e1cff
I20260501 13:58:45.268311 4733 ts_tablet_manager.cc:1950] T df6234730e704c77a3a85a407f0e1cff P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 13:58:45.266294 4742 consensus_queue.cc:260] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.292886 4742 ts_tablet_manager.cc:1434] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.119s user 0.012s sys 0.000s
I20260501 13:58:45.304103 4742 tablet_bootstrap.cc:492] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:58:45.316987 3153 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]). Expected version 1 got 0
W20260501 13:58:45.318885 3153 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0c215522a3a44450922f03cf2cb7cac3 consensus state has no leader
I20260501 13:58:45.321015 4740 tablet_bootstrap.cc:492] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:45.321579 4740 ts_tablet_manager.cc:1403] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.285s user 0.015s sys 0.015s
I20260501 13:58:45.327049 4740 raft_consensus.cc:359] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.327926 4740 raft_consensus.cc:385] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:45.328275 4740 raft_consensus.cc:740] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:45.329113 4740 consensus_queue.cc:260] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.358651 4740 ts_tablet_manager.cc:1434] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.037s user 0.003s sys 0.004s
W20260501 13:58:45.347615 3141 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0c215522a3a44450922f03cf2cb7cac3 consensus state has no leader
I20260501 13:58:45.375944 4740 tablet_bootstrap.cc:492] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:58:45.378438 4742 tablet_bootstrap.cc:654] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:45.388689 4740 tablet_bootstrap.cc:654] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:58:45.414858 1090 master-stress-test.cc:577] Killing test thread 1/10
W20260501 13:58:45.436074 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:45.449185 4742 tablet_bootstrap.cc:492] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:58:45.449667 4742 ts_tablet_manager.cc:1403] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.147s user 0.013s sys 0.003s
I20260501 13:58:45.453684 4742 raft_consensus.cc:359] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.454384 4742 raft_consensus.cc:385] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:45.454697 4742 raft_consensus.cc:740] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:58:45.455354 4742 consensus_queue.cc:260] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.459923 4740 tablet_bootstrap.cc:492] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:58:45.460374 4740 ts_tablet_manager.cc:1403] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.085s user 0.010s sys 0.004s
I20260501 13:58:45.471597 4742 ts_tablet_manager.cc:1434] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.022s user 0.004s sys 0.004s
I20260501 13:58:45.470069 4740 raft_consensus.cc:359] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.472208 4740 raft_consensus.cc:385] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:58:45.472522 4740 raft_consensus.cc:740] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:58:45.476454 3153 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]). Expected version 1 got 0
I20260501 13:58:45.473227 4740 consensus_queue.cc:260] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.489326 3154 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]). Expected version 1 got 0
I20260501 13:58:45.493410 4740 ts_tablet_manager.cc:1434] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20260501 13:58:45.511751 3154 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]). Expected version 1 got 0
I20260501 13:58:45.514271 3153 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]). Expected version 1 got 0
W20260501 13:58:45.530076 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 3266ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:45.618369 4696 raft_consensus.cc:493] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:45.618891 4696 raft_consensus.cc:515] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.625085 4696 leader_election.cc:290] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:45.627722 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c215522a3a44450922f03cf2cb7cac3" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:58:45.628523 3672 raft_consensus.cc:2468] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:45.629515 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c215522a3a44450922f03cf2cb7cac3" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:45.630185 3513 raft_consensus.cc:2468] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:58:45.632998 4731 raft_consensus.cc:493] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:45.633343 3756 leader_election.cc:304] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:45.633455 4731 raft_consensus.cc:515] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.635368 4731 leader_election.cc:290] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:45.636082 3820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6087f52d19984dc2a83b6f923685610a" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:58:45.636752 3820 raft_consensus.cc:2468] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:45.638386 3608 leader_election.cc:304] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:45.638696 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6087f52d19984dc2a83b6f923685610a" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:58:45.639250 4731 raft_consensus.cc:2804] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:45.639261 3513 raft_consensus.cc:2468] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 0.
I20260501 13:58:45.639575 4731 raft_consensus.cc:493] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:45.639863 4731 raft_consensus.cc:3060] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:45.643254 4696 raft_consensus.cc:2804] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:58:45.643735 4696 raft_consensus.cc:493] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:58:45.644137 4696 raft_consensus.cc:3060] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:45.647773 4731 raft_consensus.cc:515] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.654403 3820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6087f52d19984dc2a83b6f923685610a" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:58:45.655074 3820 raft_consensus.cc:3060] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:45.672016 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6087f52d19984dc2a83b6f923685610a" candidate_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:45.672780 3513 raft_consensus.cc:3060] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:45.665875 4696 raft_consensus.cc:515] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.682830 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c215522a3a44450922f03cf2cb7cac3" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:58:45.683399 3672 raft_consensus.cc:3060] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:45.689421 3515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c215522a3a44450922f03cf2cb7cac3" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:58:45.690024 3515 raft_consensus.cc:3060] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:58:45.700634 3515 raft_consensus.cc:2468] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:45.702028 3757 leader_election.cc:304] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:58:45.702855 4751 raft_consensus.cc:2804] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:45.699549 4731 leader_election.cc:290] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Requested vote from peers 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:45.682451 4696 leader_election.cc:290] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:58:45.714275 3672 raft_consensus.cc:2468] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:58:45.717021 3820 raft_consensus.cc:2468] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:45.723779 4708 mvcc.cc:204] Tried to move back new op lower bound from 7281229513157271552 to 7281229469435731968. Current Snapshot: MvccSnapshot[applied={T|T < 7281229513157271552 or (T in {7281229513157271552})}]
I20260501 13:58:45.725234 3608 leader_election.cc:304] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:58:45.726574 4731 raft_consensus.cc:2804] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:58:45.733323 4751 raft_consensus.cc:697] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:58:45.734342 4751 consensus_queue.cc:237] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:58:45.745427 3513 raft_consensus.cc:2468] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af0cedc7f3c47299b93e6e92d8e02d1 in term 1.
I20260501 13:58:45.757483 4731 raft_consensus.cc:697] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Running, Role: LEADER
I20260501 13:58:45.758955 4731 consensus_queue.cc:237] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
W20260501 13:58:45.774722 3139 leader_election.cc:336] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253): Timed out: RequestConsensusVote RPC to 127.1.16.188:38253 timed out after 1.007s (SENT)
I20260501 13:58:45.784198 3153 catalog_manager.cc:5671] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:45.786525 3153 catalog_manager.cc:5763] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685) does not have the latest schema for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]). Expected version 1 got 0
I20260501 13:58:45.808589 3154 catalog_manager.cc:5671] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130). New cstate: current_term: 1 leader_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } }
I20260501 13:58:45.995187 3672 raft_consensus.cc:1275] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Refusing update from remote peer 5c239ccbdc554619a237d78e958ffbc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:58:45.997718 3513 raft_consensus.cc:1275] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Refusing update from remote peer 5c239ccbdc554619a237d78e958ffbc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 13:58:45.999416 4696 consensus_queue.cc:1048] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:46.000816 4696 consensus_queue.cc:1048] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:58:46.031816 3154 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]). Expected version 1 got 0
I20260501 13:58:46.053244 4765 mvcc.cc:204] Tried to move back new op lower bound from 7281229520793145344 to 7281229519827243008. Current Snapshot: MvccSnapshot[applied={T|T < 7281229520793145344}]
I20260501 13:58:46.054569 4770 tablet.cc:1724] T 0c215522a3a44450922f03cf2cb7cac3 P 5c239ccbdc554619a237d78e958ffbc2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:46.081220 3152 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]). Expected version 1 got 0
I20260501 13:58:46.097704 3153 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab] alter complete (version 1)
I20260501 13:58:46.151288 4771 tablet.cc:1724] T 0c215522a3a44450922f03cf2cb7cac3 P 6af0cedc7f3c47299b93e6e92d8e02d1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:46.158936 4772 tablet.cc:1724] T 0c215522a3a44450922f03cf2cb7cac3 P 9154033cea374a10a5225c53b8bcf122: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 13:58:46.164888 3153 catalog_manager.cc:5763] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857) does not have the latest schema for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]). Expected version 1 got 0
I20260501 13:58:46.189921 3152 catalog_manager.cc:5763] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335) does not have the latest schema for tablet 0c215522a3a44450922f03cf2cb7cac3 (table default.table_995cfbea106b4656b4929310efb5017f [id=9535ed975c6d458f97c9175b0a34f3ab]). Expected version 1 got 0
I20260501 13:58:46.211326 4761 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:46.211838 4761 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:46.213798 4761 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:46.215725 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 13:58:46.216349 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:46.216665 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:46.217759 4761 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:46.247162 4761 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:46.247675 4761 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:46.249639 4761 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:46.256029 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:58:46.258134 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:58:46.258563 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:46.259985 4761 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:58:46.333128 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 3266ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:46.634826 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:46.661270 4774 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:46.670519 4774 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:46.684741 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:46.690986 4774 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:46.692699 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:46.695111 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:46.696398 4774 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 13:58:46.834859 4731 consensus_queue.cc:1048] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 13:58:46.860754 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 4996ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:46.933382 4731 consensus_queue.cc:1048] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260501 13:58:47.156674 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 4996ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:47.259075 4787 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:47.259569 4787 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:47.261478 4787 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:47.262609 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:47.263127 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:47.264657 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:47.274667 4787 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:47.614735 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 5820ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:47.677245 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:47.882143 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 5820ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:48.331905 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 6545ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:48.559247 4796 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:48.559600 4796 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:48.561025 4796 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:48.562342 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:48.565940 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
W20260501 13:58:48.566391 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 6545ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:48.567962 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:48.569576 4796 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:48.662933 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:49.011881 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 7229ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:49.158631 4802 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:49.159206 4802 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:49.195760 4802 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:49.196820 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:49.198012 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:49.200556 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:49.207090 4802 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:49.263176 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 7229ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:49.593557 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:49.699636 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 7926ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:49.943204 4807 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:49.943714 4807 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:49.945327 4807 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:49.947173 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 13:58:49.947674 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:58:49.947973 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:49.948470 4807 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:58:49.992823 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 7926ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:50.460275 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 8656ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:50.666188 4812 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:50.666806 4812 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:50.668889 4812 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:50.669696 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:50.670081 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:50.672379 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:50.673424 4812 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:50.702455 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 8656ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:50.829888 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:51.132375 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 9365ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:51.251959 4817 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:51.252519 4817 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:51.255216 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:51.255458 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:51.256887 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:51.254402 4817 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:51.260704 4818 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:51.392942 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 9365ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:51.398404 4038 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b9f3b4229594e13959fec0aa8281c03 consensus state has no leader
W20260501 13:58:51.839170 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 10056ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:52.053525 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:52.084745 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 10056ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:52.391292 4822 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:52.391819 4822 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:52.393703 4822 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:52.396490 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 13:58:52.397104 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:58:52.397462 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:52.398097 4822 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:58:52.526816 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 10748ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:52.777942 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 10748ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:53.093513 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:53.095762 4827 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:53.099694 4827 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:53.101722 4827 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:53.102757 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:53.104485 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:53.106647 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:53.107467 4827 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:53.218509 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 11441ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:53.468904 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 11441ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:53.666893 4831 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:53.667402 4831 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:53.669970 4831 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:53.671109 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:53.671605 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:53.672797 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:53.673399 4831 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:53.872548 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:53.927170 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 12132ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:54.154016 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 12132ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:54.169126 4833 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:54.169617 4833 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:54.171568 4833 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:54.176411 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:58:54.177428 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:58:54.177804 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:54.179574 4826 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:54.218406 4833 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:58:54.218915 4833 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:58:54.220764 4833 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:58:54.223176 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 13:58:54.224140 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:58:54.224491 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:58:54.225255 4833 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:58:54.256505 4835 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:54.256999 4835 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:54.258942 4835 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:54.259967 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:54.260432 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:54.261931 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:54.263490 4835 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:54.612673 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 12817ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:54.879384 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 12817ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:54.985960 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 11: this message will repeat every 5th retry.
W20260501 13:58:55.000226 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:55.258322 4841 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:55.259104 4841 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:55.261307 4841 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:55.262291 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:55.264405 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:55.268431 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:55.269229 4841 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:55.340301 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 13543ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:55.587574 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 13543ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:55.796077 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:55.985827 3557 debug-util.cc:398] Leaking SignalData structure 0x7b0800121f20 after lost signal to thread 3422
W20260501 13:58:55.988227 3557 debug-util.cc:398] Leaking SignalData structure 0x7b08001279a0 after lost signal to thread 3560
W20260501 13:58:56.041419 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 14251ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:56.315819 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 14251ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:56.326512 4849 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:56.327054 4849 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:56.330775 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:56.334185 4849 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:56.334901 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:56.336031 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:56.336869 4849 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:56.793516 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 14979ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:58:56.918898 4851 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:56.919763 4851 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:56.922703 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:56.925853 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:56.927445 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:56.932555 4851 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:56.933632 4851 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:57.058305 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:57.068487 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 14979ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:57.541189 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 15731ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:57.814512 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 15731ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:57.921566 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:58:57.953538 4861 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:58:57.954058 4861 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:58:57.955900 4861 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:58:57.956656 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:58:57.956998 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:58:57.958235 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:58:57.958853 4861 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:58:58.337632 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 16477ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:58.624051 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 16477ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:58.744315 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:59.054594 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 17287ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:59.300710 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 17287ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:58:59.725852 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:58:59.757050 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 17964ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:00.014638 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 17964ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:00.468176 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 18678ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:00.547763 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:00.716902 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 18678ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:00.735232 4873 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:00.735733 4873 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:00.738385 4873 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:00.739092 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:00.739455 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:00.741093 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:00.741886 4873 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:01.167133 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 19380ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:01.368916 4871 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:01.369449 4871 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:01.372597 4871 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:01.374747 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 13:59:01.375624 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:59:01.375911 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:01.376646 4871 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:01.426772 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 19380ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:01.470501 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:01.669898 4876 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:01.670485 4876 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:01.672444 4876 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:01.673195 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:01.673493 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:01.675256 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:01.675999 4876 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:01.896966 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 20090ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:02.149129 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 20090ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:02.224905 4879 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:02.225445 4879 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:02.228093 4879 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:02.228933 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:02.229121 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:02.230619 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:02.231374 4879 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 13:59:02.301102 4859 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:02.301625 4859 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:02.303646 4859 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:02.305259 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:59:02.305879 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:59:02.306236 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:02.306937 4859 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:02.415803 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:02.581615 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 20812ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:02.827872 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 20812ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:02.860859 4883 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:02.861948 4883 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:02.863878 4883 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:02.864610 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:02.864949 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:02.866176 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:02.866950 4883 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:03.289510 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 21491ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:03.344852 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:03.403700 4888 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:03.404388 4888 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:03.406709 4888 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:03.407222 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:03.410858 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:03.413225 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:03.414008 4888 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:03.538450 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 21491ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:04.003589 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 22201ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:04.280074 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 22201ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:04.307543 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 16: this message will repeat every 5th retry.
W20260501 13:59:04.311901 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:04.751845 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 22943ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:05.008772 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 22943ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:05.125128 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:05.436359 4891 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:05.436863 4891 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:05.438683 4891 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:05.440470 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 13:59:05.441097 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:59:05.441455 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:05.442346 4891 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:05.461001 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 23672ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:05.696169 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 23672ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:05.839152 4900 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:05.839831 4900 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:05.841828 4900 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:05.842634 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:05.843428 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:05.845328 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:05.846891 4900 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:05.922631 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:06.143543 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 24359ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:06.412424 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 24359ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:06.867520 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:06.870162 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 25075ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:07.119827 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 25075ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:07.226042 4907 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:07.226508 4907 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:07.228294 4907 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:07.229544 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:07.229600 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:07.231019 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:07.231734 4907 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:07.548941 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 25783ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:07.778730 4909 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:07.779670 4909 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:07.783324 4909 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:07.784420 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:07.785557 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:07.787007 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:07.788383 4909 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:07.795218 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 25783ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:07.796480 4038 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b9f3b4229594e13959fec0aa8281c03 consensus state has no leader
W20260501 13:59:07.896593 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:08.244392 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 26458ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:08.497390 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 26458ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:08.768355 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:08.815333 4918 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:08.815918 4918 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:08.818081 4918 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:08.819268 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:08.820679 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:08.822335 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:08.823020 4918 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:08.967846 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 27160ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:09.193114 4914 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:09.193749 4914 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:09.196022 4914 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:09.199688 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:59:09.200340 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:59:09.200672 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:09.201747 4914 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:09.217049 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 27160ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:09.549566 4914 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:09.550066 4914 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:09.552017 4914 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:09.554605 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 13:59:09.555249 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:59:09.555570 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:09.556598 4914 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:09.666549 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 27880ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:09.925560 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 27880ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:09.972361 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:10.307672 4924 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:10.308599 4924 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:10.312381 4924 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:10.312834 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:10.314968 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:10.316393 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:10.318197 4924 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:10.385450 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 28589ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:10.631693 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 28589ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:10.907320 4927 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:10.909296 4927 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:10.913384 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:10.917915 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:10.919509 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:10.922636 4927 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:10.923622 4927 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:11.027315 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:11.090301 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 29295ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:11.342933 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 29295ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:11.793515 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 30006ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:11.932292 4936 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:11.932878 4936 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:11.936303 4936 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:11.937450 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:11.937407 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:11.939311 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:11.940135 4936 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:11.970598 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:12.050431 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 30006ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:12.498924 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 30713ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:12.730193 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 30713ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:12.926668 4941 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:12.927212 4941 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:12.929863 4941 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:12.929837 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:12.930567 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:12.931859 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:12.932953 4941 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:13.177191 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:13.220338 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 31393ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:13.452198 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 31393ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:13.501153 4943 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:13.501652 4943 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:13.503607 4943 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:13.505035 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:59:13.505822 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:59:13.506101 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:13.506731 4943 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:13.908054 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 32115ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:13.933622 4947 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:13.942503 4947 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:13.946334 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:13.950477 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:13.952066 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:13.959147 4947 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:13.960155 4947 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:14.117413 3139 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.1.16.188:38253 timed out after 30.000s (SENT) [suppressed 3 similar messages]
W20260501 13:59:14.163842 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 32115ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:14.360481 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 21: this message will repeat every 5th retry.
W20260501 13:59:14.382321 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:14.619222 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 32827ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:14.889145 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 32827ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:15.220026 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:15.342906 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 33553ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:15.358693 4958 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:15.359404 4958 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:15.361763 4958 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:15.362627 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:15.363597 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:15.365612 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:15.366530 4958 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:15.399837 3185 debug-util.cc:398] Leaking SignalData structure 0x7b08000a47a0 after lost signal to thread 3122
W20260501 13:59:15.400558 3185 debug-util.cc:398] Leaking SignalData structure 0x7b08000a5820 after lost signal to thread 3188
W20260501 13:59:15.610157 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 33553ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:15.948560 4961 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:15.951274 4961 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:15.959113 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:15.961798 4961 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:15.962848 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:15.964577 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:15.972043 4961 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:16.099084 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:16.104972 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 34273ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:16.403985 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 34273ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:16.891577 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 35067ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:16.926651 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:17.202178 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 35067ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:17.684782 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 35865ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:17.869874 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:17.941020 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 35865ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:18.006322 4970 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:18.007148 4970 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:18.010074 4970 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:18.011999 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 13:59:18.012614 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:59:18.012914 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:18.013558 4970 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:59:18.348951 4971 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:18.349911 4971 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:18.355158 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:18.359516 4971 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:18.359583 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:18.361956 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:18.363185 4971 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:18.400734 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 36604ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:18.636444 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 36604ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:18.656173 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:18.947391 4977 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:18.948722 4977 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:18.953385 4977 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:18.953886 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:18.956722 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:18.958276 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:18.961321 4977 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:19.082759 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 37299ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:19.348758 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 37299ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:19.805112 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 38012ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:19.904385 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:19.969133 4983 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:19.975535 4983 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:19.978436 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:19.979157 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:19.980829 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:19.982175 4983 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:19.982790 4983 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:20.065448 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 38012ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:20.402726 4975 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:20.403215 4975 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:20.405280 4975 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:20.407114 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 13:59:20.408322 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:59:20.408700 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:20.409905 4975 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:20.492762 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 38728ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:20.716864 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 38728ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:20.959607 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:21.170524 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 39380ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:21.418033 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 39380ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:21.835075 4989 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:21.835518 4989 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:21.837361 4989 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:21.838420 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:21.838863 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:21.840466 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:21.841495 4989 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:21.861640 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 40081ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:22.105644 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:22.110143 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 40081ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:22.260769 4982 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:22.261219 4982 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:22.262665 4982 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:22.264266 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:59:22.264796 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:59:22.265033 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:22.265532 4982 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:59:22.460338 4994 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:22.460883 4994 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:22.462832 4994 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:22.464306 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:22.465447 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:22.466655 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:22.467434 4994 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:22.564783 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 40773ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:22.825599 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 40773ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:23.016060 4996 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:23.016561 4996 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:23.018047 4996 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:23.019418 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:23.019557 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:23.021001 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:23.021790 4996 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:23.282317 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 41489ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:23.320804 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:23.523272 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 41489ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:23.979802 5000 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:23.980307 5000 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
W20260501 13:59:23.980697 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 42186ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:23.982800 5000 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:23.984485 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:23.984975 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:23.986354 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:23.987600 5000 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:24.035349 3719 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (30 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 13:59:24.235049 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 42186ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:24.389147 3561 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (42 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 13:59:24.480172 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 26: this message will repeat every 5th retry.
W20260501 13:59:24.485935 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:24.616199 3867 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (30 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 13:59:24.694657 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 42898ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:24.962697 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 42898ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:25.414827 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 43626ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:25.422581 5006 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:25.423230 5006 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:25.425347 5006 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:25.426213 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:25.427393 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:25.429306 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:25.430442 5006 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:25.659006 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 43626ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:25.719081 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:26.002065 5011 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:26.002806 5011 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:26.005584 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:26.011411 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:26.013046 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:26.014957 5011 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:26.019224 5012 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:26.110904 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 44322ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:26.365957 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 44322ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:26.580910 5014 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:26.581835 5014 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:26.584344 5014 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:26.585019 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:26.585318 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:26.586861 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:26.588044 5014 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:26.816847 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 45029ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:26.842451 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:27.079063 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 45029ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:27.151727 5018 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:27.152261 5018 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:27.153795 5018 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:27.154495 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:27.154845 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:27.156265 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:27.157096 5018 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:27.520638 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 45742ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:27.712705 5020 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:27.713423 5020 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:27.716228 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:27.718089 5020 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:27.719090 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:27.720275 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:27.721597 5020 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:27.768721 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 45742ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:27.898025 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:28.217264 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 46432ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:28.273068 5025 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:28.275575 5025 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:28.281476 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:28.282253 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:28.281095 5025 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:28.284045 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:28.285074 5025 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:28.460996 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 46432ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:28.921180 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 47124ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:29.039732 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:29.178557 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 47124ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:29.623636 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 47841ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:29.873904 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 47841ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:30.014676 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:30.306622 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 48537ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:30.550671 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 48537ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:30.931224 5036 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:30.931895 5036 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:30.934856 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:30.934810 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:30.933789 5036 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:30.937371 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:30.938133 5036 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:30.998556 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:31.008350 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 49214ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:31.263279 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 49214ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:31.712272 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 49926ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:31.943387 5041 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:31.943955 5041 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:31.945858 5041 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:31.947563 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:31.948885 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:31.953266 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:31.954049 5041 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:31.965546 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 49926ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:31.978881 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:32.428256 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 50629ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:32.551288 5050 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:32.551766 5050 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:32.554898 5050 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:32.556867 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:32.556862 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:32.558739 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:32.559717 5050 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:32.676105 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 50629ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:32.827044 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:33.111743 5055 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:33.112262 5055 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:33.114120 5055 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:33.115797 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:33.117003 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:33.118520 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:33.119614 5055 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:33.127918 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 51339ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:33.382555 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 51339ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:33.588001 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:33.846889 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 52046ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:34.108780 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 52046ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:34.534933 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 31: this message will repeat every 5th retry.
W20260501 13:59:34.545365 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:34.562943 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 52771ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:34.625490 5062 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:34.626029 5062 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:34.628115 5062 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:34.629254 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:34.629262 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:34.631052 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:34.639415 5062 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:34.827960 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 52771ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:35.272730 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 53491ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:35.521070 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 53491ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:35.635811 5069 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:35.636528 5069 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:35.638513 5069 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:35.639225 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:35.639968 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:35.641765 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:35.642403 5069 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:35.675945 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:35.983192 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 54184ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:36.102162 5070 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:36.102890 5070 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:36.105957 5070 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:36.109002 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:59:36.109615 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:59:36.109930 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:36.110697 5058 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 13:59:36.218807 5074 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:36.219301 5074 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:36.221191 5074 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:36.222237 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:36.223120 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:36.224283 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:36.224926 5074 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:36.249754 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 54184ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:36.422603 5070 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:36.423149 5070 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:36.424854 5070 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:36.426288 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 13:59:36.426908 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:59:36.427160 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:36.427755 5070 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:36.555979 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:36.686867 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 54913ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:36.818902 5077 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:36.819869 5077 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:36.826200 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:36.827970 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:36.830137 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:36.831508 5077 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:36.831795 5078 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:36.938759 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 54913ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:37.394577 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 55602ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:37.443023 5080 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:37.443475 5080 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:37.445307 5080 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:37.446236 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:37.446780 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:37.447952 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:37.448804 5080 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:37.641762 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 55602ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:37.642102 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:37.964792 5083 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:37.965255 5083 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:37.967058 5083 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:37.969077 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 13:59:37.970853 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:59:37.971184 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:37.972211 5083 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:38.094733 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 56305ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x47c8f5 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x4959cf __interceptor_pthread_setcancelstate at ??:?
@ 0x7f493f7d8fc0 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260501 13:59:38.256012 4055 rpcz_store.cc:267] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35438 (request call id 70) took 59999 ms (client timeout 59999 ms). Trace:
W20260501 13:59:38.256466 4055 rpcz_store.cc:269] 0501 13:58:38.256441 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:38.364706 (+108265us) service_pool.cc:224] Handling call
0501 13:58:40.024732 (+1660026us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:59:38.255777 (+58231045us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":333892,"Hive Metastore.run_cpu_time_us":385,"Hive Metastore.run_wall_time_us":207320}
I20260501 13:59:38.258283 4055 master_service.cc:954] ReplaceTablet: received request to replace tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 from {username='slave'} at 127.0.0.1:35438
W20260501 13:59:38.260281 3878 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.16.188:38253)
I20260501 13:59:38.277573 3153 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_aafd3ab3f9f44a74af41aae60cfe40fa" } modify_external_catalogs: true
I20260501 13:59:38.278599 3153 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_aafd3ab3f9f44a74af41aae60cfe40fa" } modify_external_catalogs: true
W20260501 13:59:38.341948 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 56305ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:38.456977 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:38.790421 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 57005ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:38.960686 3878 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.16.190:38561)
I20260501 13:59:38.975538 3153 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9074c6e0ff374999bcf7953f17de8b4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:59:38.987303 3153 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9074c6e0ff374999bcf7953f17de8b4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
13:59:39.017 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_9074c6e0ff374999bcf7953f17de8b4e already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at jdk.internal.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at jdk.proxy2/jdk.proxy2.$Proxy27.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
W20260501 13:59:39.020430 3153 catalog_manager.cc:2170] Already present: failed to create HMS catalog entry for table [id=d169834cdcbb4edabd0442a32cf6afde]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_9074c6e0ff374999bcf7953f17de8b4e already exists)
W20260501 13:59:39.021420 4056 rpcz_store.cc:267] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35438 (request call id 78) took 59999 ms (client timeout 59999 ms). Trace:
W20260501 13:59:39.021795 4056 rpcz_store.cc:269] 0501 13:58:39.021734 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:39.021978 (+ 244us) service_pool.cc:224] Handling call
0501 13:58:40.968572 (+1946594us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0501 13:59:39.021185 (+58052613us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":90,"Hive Metastore.run_cpu_time_us":1846,"Hive Metastore.run_wall_time_us":501959}
I20260501 13:59:39.023535 4056 master_service.cc:954] ReplaceTablet: received request to replace tablet 6309ec735c424460ad5a74c6b7ec0d8a from {username='slave'} at 127.0.0.1:35438
W20260501 13:59:39.023797 3878 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.16.190:38561)
W20260501 13:59:39.024214 4056 rpcz_store.cc:267] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:35438 (request call id 98) took 57377 ms (client timeout 59999 ms). Trace:
W20260501 13:59:39.024542 4056 rpcz_store.cc:269] 0501 13:58:41.646913 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:59:39.023349 (+57376436us) service_pool.cc:224] Handling call
0501 13:59:39.024143 (+ 794us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260501 13:59:39.034078 3153 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f701ba5ffc3845bf8cb5e55102e60688" } new_table_name: "default.Table_3240c40926184fd7b92c22b821de47e9" modify_external_catalogs: true
I20260501 13:59:39.036971 4056 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:56550 (request call id 0) took 55632 ms. Trace:
I20260501 13:59:39.037424 4056 rpcz_store.cc:276] 0501 13:58:43.404299 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:59:39.035879 (+55631580us) service_pool.cc:224] Handling call
0501 13:59:39.036809 (+ 930us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 13:59:39.038839 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 57005ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:39.487756 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 57702ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:39.620087 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:39.655104 5099 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:39.656045 5099 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:39.661744 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:39.662832 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 217 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:39.665263 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:39.667263 5099 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:39.667738 5100 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:39.728466 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 57702ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:40.173808 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 58391ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:40.275233 4066 rpcz_store.cc:267] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 85) took 59999 ms (client timeout 59999 ms). Trace:
W20260501 13:59:40.275655 4066 rpcz_store.cc:269] 0501 13:58:40.275491 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:40.275628 (+ 137us) service_pool.cc:224] Handling call
0501 13:59:40.275011 (+59999383us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 13:59:40.279110 3878 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.16.190:38561)
I20260501 13:59:40.286361 4066 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35438:
name: "default.Table_8f8971cc622d4dca8d21a3656e765390"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:59:40.426038 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 58391ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:40.430097 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:40.586191 4038 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b9f3b4229594e13959fec0aa8281c03 consensus state has no leader
W20260501 13:59:40.614836 4054 rpcz_store.cc:267] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 89) took 59999 ms (client timeout 59999 ms). Trace:
W20260501 13:59:40.615199 4054 rpcz_store.cc:269] 0501 13:58:40.615201 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:40.791767 (+176566us) service_pool.cc:224] Handling call
0501 13:58:40.816944 (+ 25177us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b5400030554. stack: 00007f4936204c83 00007f4936204b80 00007f4935d75574 000000000050e094 000000000050e029 00007f493d254470 00007f493d2f331a 00007f493f00746c 00007f493f102936 00007f493cd0f524 00007f493cd0f4b2 00007f493cd0f3e1 00007f493cd0f35c 00007f493cd0e612 00007f493afddbcc 00007f493afdd006
0501 13:59:40.614645 (+59797701us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55970176}
I20260501 13:59:40.621472 3153 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_96b1c2cf6e7c4d82a665a14d3decdab3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 13:59:40.629554 3153 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_96b1c2cf6e7c4d82a665a14d3decdab3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 13:59:40.674026 3153 catalog_manager.cc:2174] created HMS catalog entry for table [id=4adaf7985539491b851a2c1ba00f98a7]
I20260501 13:59:40.752208 3800 tablet_service.cc:1511] Processing CreateTablet for tablet 6459a231fe7946b2b4ae6e96df5e2adf (DEFAULT_TABLE table=default.table_96b1c2cf6e7c4d82a665a14d3decdab3 [id=4adaf7985539491b851a2c1ba00f98a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:59:40.753101 3651 tablet_service.cc:1511] Processing CreateTablet for tablet 6459a231fe7946b2b4ae6e96df5e2adf (DEFAULT_TABLE table=default.table_96b1c2cf6e7c4d82a665a14d3decdab3 [id=4adaf7985539491b851a2c1ba00f98a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:59:40.753577 3800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6459a231fe7946b2b4ae6e96df5e2adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:59:40.754406 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6459a231fe7946b2b4ae6e96df5e2adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:59:40.760156 3491 tablet_service.cc:1511] Processing CreateTablet for tablet 6459a231fe7946b2b4ae6e96df5e2adf (DEFAULT_TABLE table=default.table_96b1c2cf6e7c4d82a665a14d3decdab3 [id=4adaf7985539491b851a2c1ba00f98a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:59:40.761421 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6459a231fe7946b2b4ae6e96df5e2adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:59:40.787441 5111 tablet_bootstrap.cc:492] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:59:40.798034 5113 tablet_bootstrap.cc:492] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:59:40.807096 5112 tablet_bootstrap.cc:492] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 13:59:40.808795 5113 tablet_bootstrap.cc:654] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 13:59:40.807801 5111 tablet_bootstrap.cc:654] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:59:40.820979 5112 tablet_bootstrap.cc:654] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:59:40.828013 5113 tablet_bootstrap.cc:492] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:59:40.828833 5113 ts_tablet_manager.cc:1403] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.000s
I20260501 13:59:40.832249 5113 raft_consensus.cc:359] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:40.833377 5113 raft_consensus.cc:385] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:59:40.833791 5113 raft_consensus.cc:740] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:59:40.834697 5113 consensus_queue.cc:260] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:40.840178 5112 tablet_bootstrap.cc:492] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:59:40.840705 5112 ts_tablet_manager.cc:1403] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.005s
I20260501 13:59:40.850621 5112 raft_consensus.cc:359] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:40.851403 5112 raft_consensus.cc:385] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:59:40.851742 5112 raft_consensus.cc:740] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:59:40.852576 5112 consensus_queue.cc:260] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:40.854329 5111 tablet_bootstrap.cc:492] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:59:40.855512 5111 ts_tablet_manager.cc:1403] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.069s user 0.014s sys 0.014s
I20260501 13:59:40.858860 5112 ts_tablet_manager.cc:1434] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20260501 13:59:40.862178 5113 ts_tablet_manager.cc:1434] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.033s user 0.004s sys 0.003s
I20260501 13:59:40.883723 5111 raft_consensus.cc:359] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:40.900238 5111 raft_consensus.cc:385] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:59:40.900975 5111 raft_consensus.cc:740] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:59:40.902596 5111 consensus_queue.cc:260] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
W20260501 13:59:40.928306 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 59089ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:40.930213 5111 ts_tablet_manager.cc:1434] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.074s user 0.008s sys 0.005s
I20260501 13:59:40.956243 5103 raft_consensus.cc:493] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:40.956748 5103 raft_consensus.cc:515] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:40.959877 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6459a231fe7946b2b4ae6e96df5e2adf" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:59:40.960510 3672 raft_consensus.cc:2468] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 13:59:40.961853 3448 leader_election.cc:304] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:59:40.963054 5103 leader_election.cc:290] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:59:40.963585 5103 raft_consensus.cc:2804] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:59:40.963873 5103 raft_consensus.cc:493] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:59:40.964143 5103 raft_consensus.cc:3060] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:59:40.971276 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6459a231fe7946b2b4ae6e96df5e2adf" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 13:59:40.971937 3822 raft_consensus.cc:2468] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
W20260501 13:59:40.975764 4061 rpcz_store.cc:267] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 92) took 59999 ms (client timeout 59999 ms). Trace:
W20260501 13:59:40.976130 4061 rpcz_store.cc:269] 0501 13:58:40.976136 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:40.977434 (+ 1298us) service_pool.cc:224] Handling call
0501 13:59:40.975539 (+59998105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260501 13:59:40.976447 5103 raft_consensus.cc:515] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
W20260501 13:59:40.982820 3722 tablet.cc:2404] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:59:40.987274 3153 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f3eecd105941437fbf670fab2685bcf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 13:59:40.990166 5103 leader_election.cc:290] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 13:59:40.991482 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6459a231fe7946b2b4ae6e96df5e2adf" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:59:40.991706 3822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6459a231fe7946b2b4ae6e96df5e2adf" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 13:59:40.992007 3672 raft_consensus.cc:3060] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:59:40.992236 3822 raft_consensus.cc:3060] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:59:41.001914 3822 raft_consensus.cc:2468] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:59:41.002329 3672 raft_consensus.cc:2468] T 6459a231fe7946b2b4ae6e96df5e2adf P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 13:59:41.003307 3448 leader_election.cc:304] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:59:41.004532 5103 raft_consensus.cc:2804] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:59:41.005002 5103 raft_consensus.cc:697] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 13:59:41.008181 5103 consensus_queue.cc:237] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
W20260501 13:59:41.013638 3153 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f3eecd105941437fbf670fab2685bcf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260501 13:59:41.016240 3568 tablet.cc:2404] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:59:41.034845 3152 catalog_manager.cc:5671] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9154033cea374a10a5225c53b8bcf122 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "9154033cea374a10a5225c53b8bcf122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: HEALTHY } } }
I20260501 13:59:41.060403 3153 catalog_manager.cc:2174] created HMS catalog entry for table [id=dc41bd3cdd29407b9375bdf67c0397ff]
I20260501 13:59:41.099184 3651 tablet_service.cc:1511] Processing CreateTablet for tablet 43279a78f8ae439ca1323965587a3e69 (DEFAULT_TABLE table=default.table_f3eecd105941437fbf670fab2685bcf6 [id=dc41bd3cdd29407b9375bdf67c0397ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:59:41.100195 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43279a78f8ae439ca1323965587a3e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:59:41.108558 3800 tablet_service.cc:1511] Processing CreateTablet for tablet 43279a78f8ae439ca1323965587a3e69 (DEFAULT_TABLE table=default.table_f3eecd105941437fbf670fab2685bcf6 [id=dc41bd3cdd29407b9375bdf67c0397ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:59:41.109633 3800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43279a78f8ae439ca1323965587a3e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:59:41.113440 3491 tablet_service.cc:1511] Processing CreateTablet for tablet 43279a78f8ae439ca1323965587a3e69 (DEFAULT_TABLE table=default.table_f3eecd105941437fbf670fab2685bcf6 [id=dc41bd3cdd29407b9375bdf67c0397ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 13:59:41.115377 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43279a78f8ae439ca1323965587a3e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 13:59:41.122707 5111 tablet_bootstrap.cc:492] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 13:59:41.128758 5111 tablet_bootstrap.cc:654] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 13:59:41.148478 5111 tablet_bootstrap.cc:492] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 13:59:41.148921 5111 ts_tablet_manager.cc:1403] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.005s
I20260501 13:59:41.151670 5111 raft_consensus.cc:359] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:41.152384 5111 raft_consensus.cc:385] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:59:41.152693 5111 raft_consensus.cc:740] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 13:59:41.153523 5111 consensus_queue.cc:260] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:41.156608 5113 tablet_bootstrap.cc:492] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 13:59:41.157514 5112 tablet_bootstrap.cc:492] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
W20260501 13:59:41.160396 3872 tablet.cc:2404] T 6459a231fe7946b2b4ae6e96df5e2adf P 5c239ccbdc554619a237d78e958ffbc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 13:59:41.166093 5111 ts_tablet_manager.cc:1434] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.017s user 0.001s sys 0.006s
I20260501 13:59:41.180486 5113 tablet_bootstrap.cc:654] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
W20260501 13:59:41.183615 4064 rpcz_store.cc:267] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35438 (request call id 94) took 59999 ms (client timeout 59999 ms). Trace:
W20260501 13:59:41.183995 4064 rpcz_store.cc:269] 0501 13:58:41.184142 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:41.184258 (+ 116us) service_pool.cc:224] Handling call
0501 13:59:41.183383 (+59999125us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 13:59:41.186151 4063 rpcz_store.cc:267] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35438 (request call id 95) took 60001 ms (client timeout 59999 ms). Trace:
W20260501 13:59:41.186518 4063 rpcz_store.cc:269] 0501 13:58:41.184893 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:58:41.242622 (+ 57729us) service_pool.cc:224] Handling call
0501 13:59:41.185949 (+59943327us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 13:59:41.189679 3878 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.16.190:38561)
W20260501 13:59:41.190505 3878 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.16.190:38561)
I20260501 13:59:41.191957 5112 tablet_bootstrap.cc:654] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 13:59:41.197489 4063 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_124cdf87baa148fbbc6e5f06d1efcda1" } new_table_name: "default.Table_83691904199e4bcaa0dd9732f655dec6" modify_external_catalogs: true
I20260501 13:59:41.199254 4064 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_0da08f161093429e83b826825c3a9727" } modify_external_catalogs: true
I20260501 13:59:41.200038 4064 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_0da08f161093429e83b826825c3a9727" } modify_external_catalogs: true
I20260501 13:59:41.223445 5112 tablet_bootstrap.cc:492] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 13:59:41.224467 5112 ts_tablet_manager.cc:1403] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.003s
I20260501 13:59:41.233668 5112 raft_consensus.cc:359] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:41.242511 5113 tablet_bootstrap.cc:492] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 13:59:41.243427 5113 ts_tablet_manager.cc:1403] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.087s user 0.016s sys 0.006s
I20260501 13:59:41.243793 5112 raft_consensus.cc:385] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:59:41.245913 5112 raft_consensus.cc:740] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 13:59:41.251305 5112 consensus_queue.cc:260] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:41.267273 5112 ts_tablet_manager.cc:1434] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.042s user 0.010s sys 0.000s
I20260501 13:59:41.263655 5113 raft_consensus.cc:359] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:41.268237 5113 raft_consensus.cc:385] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 13:59:41.268571 5113 raft_consensus.cc:740] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 13:59:41.269227 5113 consensus_queue.cc:260] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:41.271394 5113 ts_tablet_manager.cc:1434] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.027s user 0.001s sys 0.006s
W20260501 13:59:41.320390 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 59089ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:41.478888 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:41.517352 3878 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.16.188:38253)
I20260501 13:59:41.521502 4061 master_service.cc:954] ReplaceTablet: received request to replace tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 from {username='slave'} at 127.0.0.1:35438
I20260501 13:59:41.557130 5117 raft_consensus.cc:493] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:41.557641 5117 raft_consensus.cc:515] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:41.560926 5117 leader_election.cc:290] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:59:41.560873 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43279a78f8ae439ca1323965587a3e69" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 13:59:41.561229 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43279a78f8ae439ca1323965587a3e69" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122" is_pre_election: true
I20260501 13:59:41.561491 3672 raft_consensus.cc:2468] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:59:41.561748 3513 raft_consensus.cc:2468] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 0.
I20260501 13:59:41.562794 3756 leader_election.cc:304] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1; no voters:
I20260501 13:59:41.564327 5117 raft_consensus.cc:2804] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 13:59:41.564715 5117 raft_consensus.cc:493] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 13:59:41.565073 5117 raft_consensus.cc:3060] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:59:41.572300 5117 raft_consensus.cc:515] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:41.577844 3672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43279a78f8ae439ca1323965587a3e69" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 13:59:41.578393 3672 raft_consensus.cc:3060] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:59:41.578776 5117 leader_election.cc:290] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335)
I20260501 13:59:41.578981 3513 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43279a78f8ae439ca1323965587a3e69" candidate_uuid: "5c239ccbdc554619a237d78e958ffbc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9154033cea374a10a5225c53b8bcf122"
I20260501 13:59:41.579495 3513 raft_consensus.cc:3060] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 13:59:41.585031 3513 raft_consensus.cc:2468] T 43279a78f8ae439ca1323965587a3e69 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:59:41.585595 3672 raft_consensus.cc:2468] T 43279a78f8ae439ca1323965587a3e69 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c239ccbdc554619a237d78e958ffbc2 in term 1.
I20260501 13:59:41.586295 3757 leader_election.cc:304] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 13:59:41.589445 5117 raft_consensus.cc:2804] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 13:59:41.597889 5117 raft_consensus.cc:697] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Running, Role: LEADER
I20260501 13:59:41.598851 5117 consensus_queue.cc:237] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 13:59:41.610044 3152 catalog_manager.cc:5671] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131). New cstate: current_term: 1 leader_uuid: "5c239ccbdc554619a237d78e958ffbc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: UNKNOWN } } }
W20260501 13:59:41.780388 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 59983ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:42.035974 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 59983ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:42.041684 5103 consensus_queue.cc:1048] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:59:42.081823 5134 consensus_queue.cc:1048] T 6459a231fe7946b2b4ae6e96df5e2adf P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 13:59:42.385748 5117 consensus_queue.cc:1048] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260501 13:59:42.398386 5141 consensus_queue.cc:1048] T 43279a78f8ae439ca1323965587a3e69 P 5c239ccbdc554619a237d78e958ffbc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260501 13:59:42.444857 3153 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:42.512260 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 60699ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:42.760460 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 60699ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:42.935271 5144 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:42.935757 5144 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:42.937680 5144 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:42.938403 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:42.938763 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:42.943254 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:42.943835 5144 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:43.218137 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 61424ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:43.470677 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 61424ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:43.604575 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:43.846379 5150 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:43.847213 5150 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:43.849090 5150 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:43.849803 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:43.850144 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:43.851403 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:43.852116 5150 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:43.930933 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 62133ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:44.123212 3139 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016 -> Peer 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253): Couldn't send request to peer 6b0f8911dd5d431d9d8d3124a4456262. Status: Timed out: UpdateConsensus RPC to 127.1.16.188:38253 timed out after 30.000s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260501 13:59:44.180647 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 62133ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:44.366420 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:44.367307 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 36: this message will repeat every 5th retry.
W20260501 13:59:44.618539 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 62843ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:44.889747 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 62843ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:45.238667 5153 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:45.239166 5153 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:45.240995 5153 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:45.244381 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 13:59:45.244998 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 13:59:45.245309 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:45.246357 5153 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:45.323036 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:45.330935 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 63553ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:45.584849 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 63553ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:46.024755 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 64248ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:46.122152 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:46.280682 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 64248ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:46.646626 5161 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:46.647253 5161 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:46.649309 5161 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:46.650837 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:46.652323 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:46.654374 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:46.655555 5161 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:46.733945 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 64944ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:47.007193 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 64944ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:47.179735 5163 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:47.180306 5163 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:47.182121 5163 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:47.184449 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:47.185518 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:47.187024 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:47.187592 5163 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:47.338871 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:47.470011 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 65670ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:47.719472 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 65670ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:48.156042 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 66382ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:48.184383 5167 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:48.184914 5167 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:48.188017 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:48.196669 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:48.198290 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:48.207295 5167 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:48.207301 5169 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:48.228838 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:48.399592 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 66382ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:48.856112 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 67062ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:49.110137 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 67062ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:49.417495 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:49.511845 5177 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:49.512493 5177 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:49.514911 5177 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:49.515914 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:49.515872 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:49.517685 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:49.518383 5177 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:49.553541 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 67773ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:49.814266 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 67773ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:50.192423 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:50.271178 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 68477ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:50.447098 5179 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:50.447868 5179 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:50.450649 5179 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:50.452494 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 13:59:50.453022 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 13:59:50.453282 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:50.453845 5179 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:50.516688 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 68477ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:50.588303 5181 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:50.588792 5181 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:50.591063 5181 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:50.591995 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:50.593155 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:50.594388 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:50.595101 5181 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:50.974136 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 69180ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:51.166831 5184 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:51.167423 5184 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:51.170817 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:51.170847 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:51.172484 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:51.169767 5184 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:51.173868 5184 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:51.234872 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 69180ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:51.274150 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:51.685289 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 69898ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:51.948067 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 69898ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:52.199085 5192 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:52.199606 5192 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:52.201572 5192 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:52.203155 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:52.203323 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:52.204787 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:52.205484 5192 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:52.412523 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 70611ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:52.520008 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:52.656878 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 70611ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:53.180873 5194 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:53.181595 5194 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:53.184343 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:53.187572 5194 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:53.195294 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:53.196710 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:53.197466 5194 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:53.314875 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 71320ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:53.364778 5179 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 13:59:53.365280 5179 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 13:59:53.367727 5179 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 13:59:53.369514 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 13:59:53.370280 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 13:59:53.370688 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 13:59:53.371376 5179 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 13:59:53.500501 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:53.547408 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 71320ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:53.818053 5200 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:53.819219 5200 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:53.823885 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:53.823601 5200 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:53.825873 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:53.827587 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:53.828179 5200 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:53.983875 4110 debug-util.cc:398] Leaking SignalData structure 0x7b080007fc80 after lost signal to thread 3931
W20260501 13:59:53.984592 4110 debug-util.cc:398] Leaking SignalData structure 0x7b0800075ce0 after lost signal to thread 4114
W20260501 13:59:54.005226 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 72210ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:54.274130 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 72210ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:54.350705 5202 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:54.351634 5202 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:54.353946 5202 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:54.354709 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:54.354984 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:54.356150 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:54.356906 5202 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:54.526782 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 41: this message will repeat every 5th retry.
W20260501 13:59:54.532644 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:54.743571 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 72937ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:54.961869 5208 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:54.963897 5208 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:54.973891 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:54.981277 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:54.993238 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:54.997134 5208 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:54.997684 5208 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:55.064172 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 72937ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:55.296607 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 13:59:55.501906 5210 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:55.502451 5210 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:55.504428 5210 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:55.506892 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:55.508128 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:55.512615 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:55.513702 5210 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:55.567354 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 73728ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:55.863384 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 73728ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:56.355854 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 74526ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 13:59:56.404635 5216 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:56.405206 5216 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:56.407348 5216 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:56.408365 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:56.408602 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:56.413269 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:56.415616 5216 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:56.464094 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:56.628476 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 74526ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:57.164716 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 75292ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:57.298112 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:57.440762 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 75292ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:57.652966 4617 debug-util.cc:398] Leaking SignalData structure 0x7b080007cf40 after lost signal to thread 4319
W20260501 13:59:57.653673 4617 debug-util.cc:398] Leaking SignalData structure 0x7b080009ddc0 after lost signal to thread 4620
W20260501 13:59:57.904047 3557 debug-util.cc:398] Leaking SignalData structure 0x7b080011d000 after lost signal to thread 3422
W20260501 13:59:57.905366 3557 debug-util.cc:398] Leaking SignalData structure 0x7b08000b25a0 after lost signal to thread 3560
W20260501 13:59:57.907790 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 76104ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:58.086915 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:58.180617 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 76104ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:58.204895 3756 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.130:45857, user_credentials={real_user=slave}} blocked reactor thread for 53426.9us
W20260501 13:59:58.211028 3448 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.16.130:45857, user_credentials={real_user=slave}} blocked reactor thread for 68529.7us
I20260501 13:59:58.355331 5228 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:58.355850 5228 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:58.357704 5228 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:58.359628 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:58.359670 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:58.363899 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:58.364871 5228 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 13:59:58.656953 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 76844ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:58.941649 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 76844ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:59.167411 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 13:59:59.414758 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 77605ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 13:59:59.713527 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 77605ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x47c8f5 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x4959cf __interceptor_pthread_setcancelstate at ??:?
@ 0x7f493f7d8ffc boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
I20260501 13:59:59.795220 5237 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 13:59:59.795864 5237 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 13:59:59.797807 5237 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 13:59:59.799325 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 13:59:59.804347 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 13:59:59.811347 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 13:59:59.814128 5237 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:00.145445 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:00.168650 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 78377ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:00.438308 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 78377ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:00.905306 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 79102ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:01.158850 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 79102ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:01.272423 5243 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:01.273332 5243 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:01.276571 5243 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:01.277240 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:01.280901 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:01.282121 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:01.283030 5243 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:01.309633 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:01.607560 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 79822ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:01.855896 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 79822ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:02.237762 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:02.319092 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 80519ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:02.508441 5244 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:02.509024 5244 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:02.511117 5244 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:02.512941 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:00:02.513552 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:00:02.513875 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:02.514554 5244 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:02.570385 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 80519ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:02.663897 5253 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:02.664839 5253 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:02.668799 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:02.671411 5253 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:02.672422 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:02.673698 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:02.674656 5253 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:03.020635 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 81233ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:03.269006 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 81233ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:03.454504 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:03.593943 5256 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:03.594409 5256 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:03.596405 5256 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:03.599663 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:00:03.600253 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:00:03.600584 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:03.601230 5256 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 14:00:03.697070 5259 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:03.698052 5259 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:03.704897 5259 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:03.705569 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:03.706777 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:03.708336 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:03.709111 5259 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:03.723538 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 81932ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:03.969277 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 81932ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:04.426926 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 82632ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:04.570861 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 46: this message will repeat every 5th retry.
W20260501 14:00:04.578958 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:04.676033 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 82632ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:04.759068 5266 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:04.759684 5266 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:04.762439 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:04.763018 5266 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:04.763773 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:04.764995 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:04.765794 5266 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:00:04.976954 5264 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:04.977466 5264 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:04.979225 5264 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:04.982250 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:00:04.982924 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:00:04.983273 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:04.984308 5264 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:05.132411 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 83339ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:05.391250 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 83339ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:05.722266 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:05.829691 5274 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:05.830542 5274 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:05.834055 5274 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:05.834031 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:05.835402 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:05.837601 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:05.838418 5274 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:05.864782 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 84054ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:06.117082 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 84054ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:06.422765 5278 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:06.423612 5278 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:06.427393 5278 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:06.427742 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:06.428901 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:06.430701 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:06.431883 5278 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:06.475710 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:06.624464 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 84780ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:06.870663 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 84780ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:07.298264 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:07.315773 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 85537ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:07.444446 3715 debug-util.cc:398] Leaking SignalData structure 0x7b080010d060 after lost signal to thread 3572
W20260501 14:00:07.445626 3715 debug-util.cc:398] Leaking SignalData structure 0x7b080011d900 after lost signal to thread 3718
W20260501 14:00:07.564214 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 85537ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:07.904121 5285 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:07.905230 5285 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:07.910166 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:07.912191 5285 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:07.914702 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:07.917269 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:07.918053 5285 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:00:08.048781 5283 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:08.049381 5283 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:08.051426 5283 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:08.055512 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:00:08.056141 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:00:08.056453 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:08.058112 5283 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:08.078706 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 86227ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:08.311003 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:08.335367 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 86227ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:08.806023 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 86998ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:08.929672 5290 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:08.930859 5290 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:08.936306 5290 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:08.936849 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:08.939261 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:08.941542 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:08.942525 5290 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:09.124069 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 86998ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:09.413965 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:09.526594 5295 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:09.527554 5295 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:09.530372 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:09.531543 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:09.533102 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:09.534776 5295 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:09.535573 5295 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:09.599903 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 87787ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:09.859794 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 87787ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:10.111721 5298 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:10.122700 5298 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:10.126730 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:10.130760 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:10.132537 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:10.135077 5298 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:10.134892 5300 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:10.297555 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:10.364719 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 88523ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:10.614887 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 88523ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:11.050925 3863 debug-util.cc:398] Leaking SignalData structure 0x7b080011f800 after lost signal to thread 3730
W20260501 14:00:11.051746 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 89278ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:11.052095 3863 debug-util.cc:398] Leaking SignalData structure 0x7b080011e160 after lost signal to thread 3866
W20260501 14:00:11.313355 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 89278ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:11.341650 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:11.782308 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 89976ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:12.055576 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 89976ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:12.194203 5307 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:12.195056 5307 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
W20260501 14:00:12.200186 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:12.205245 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:12.208549 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:12.210369 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:12.211822 5307 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:12.211966 5311 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:00:12.260643 5309 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:12.261234 5309 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
W20260501 14:00:12.266141 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:00:12.267992 5309 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:12.269124 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:00:12.269462 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:12.270731 5308 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:12.557773 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 90718ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:12.718998 5311 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:12.719532 5311 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:12.721614 5311 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:12.723814 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:12.728267 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:12.731093 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:12.732141 5311 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:12.831015 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 90718ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:13.263808 5313 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:13.264881 5313 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:13.274259 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:13.274056 5313 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:13.276499 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:13.277817 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:13.278738 5313 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:13.424608 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:13.435407 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 91494ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:13.692701 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 91494ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:14.188992 3139 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.1.16.188:38253 timed out after 30.000s (SENT)
I20260501 14:00:14.318292 5322 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:14.318969 5322 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:14.321873 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:14.325223 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:14.326901 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:14.370028 5324 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:00:14.370775 5322 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
W20260501 14:00:14.371692 4502 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.16.188:38253, user_credentials={real_user=slave}} blocked reactor thread for 44877us
W20260501 14:00:14.485185 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 51: this message will repeat every 5th retry.
W20260501 14:00:14.493661 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:14.685736 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 92356ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:14.851094 3143 maintenance_manager.cc:419] P 0a1d2cd221924677a3ef81b081e5e016: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033335
W20260501 14:00:14.949467 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 92356ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:14.976107 3142 maintenance_manager.cc:643] P 0a1d2cd221924677a3ef81b081e5e016: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.124s user 0.100s sys 0.015s Metrics: {"bytes_written":37948,"cfile_init":1,"drs_written":1,"lbm_read_time_us":298,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2712,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":98,"thread_start_us":614,"threads_started":1,"wal-append.queue_time_us":935}
I20260501 14:00:14.978626 3143 maintenance_manager.cc:419] P 0a1d2cd221924677a3ef81b081e5e016: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7240 bytes on disk
I20260501 14:00:14.980062 3142 maintenance_manager.cc:643] P 0a1d2cd221924677a3ef81b081e5e016: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":181,"lbm_reads_lt_1ms":4}
W20260501 14:00:15.088801 3715 debug-util.cc:398] Leaking SignalData structure 0x7b080011cce0 after lost signal to thread 3572
W20260501 14:00:15.089772 3715 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad660 after lost signal to thread 3718
W20260501 14:00:15.403755 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 93613ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:15.540205 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:15.661659 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 93613ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:16.099998 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 94325ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:16.359364 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 94325ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:16.549320 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:16.804603 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 95022ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:17.036931 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 95022ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:17.304955 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:17.486047 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 95700ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:17.666157 5339 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:17.667016 5339 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:17.669256 5339 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:17.670336 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:17.671569 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:17.673667 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:17.674490 5339 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:17.733551 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 95700ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:17.859687 3185 debug-util.cc:398] Leaking SignalData structure 0x7b08000a7fa0 after lost signal to thread 3122
W20260501 14:00:17.860253 3185 debug-util.cc:398] Leaking SignalData structure 0x7b080009dea0 after lost signal to thread 3188
W20260501 14:00:18.188472 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 96397ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:18.444079 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:18.458210 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 96397ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:18.741052 5342 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:18.741547 5342 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:18.744340 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:18.744979 5342 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:18.745558 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:18.747707 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:18.748875 5342 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:18.898975 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 97121ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:19.153519 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 97121ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:19.260859 5345 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:19.261379 5345 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:19.263284 5345 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:19.264312 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:19.264263 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:19.265939 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:19.266644 5345 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:19.444160 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:19.603737 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 97817ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:19.628427 5347 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:19.629027 5347 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:19.631824 5347 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:19.634351 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:00:19.634974 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:00:19.635293 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:19.635986 5347 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 14:00:19.801965 5350 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:19.802637 5350 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:19.804716 5350 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:19.805747 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:19.806151 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:19.807488 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:19.808173 5350 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:19.846930 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 97817ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:20.321708 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 98510ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:20.408859 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:20.573736 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 98510ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:20.827885 5354 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:20.828394 5354 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:20.831035 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:20.831595 5354 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:20.831758 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:20.833479 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:20.834158 5354 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:20.997608 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 99237ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:21.247701 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 99237ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:21.349617 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:21.484311 5359 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:21.484901 5359 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:21.495378 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:21.496459 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:21.498047 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:21.504246 5359 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:21.505193 5359 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:21.696343 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 99911ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:21.928260 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 99911ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:22.111523 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:22.391040 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 100591ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:22.645664 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 100591ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:22.868968 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:22.955495 5369 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:22.956004 5369 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:22.958024 5369 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:22.958798 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:22.959060 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:22.960430 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:22.961242 5369 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:23.104317 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 101309ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:23.372843 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 101309ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:23.589668 5372 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:23.590811 5372 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:23.597777 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:23.599172 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:23.599071 5372 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:23.600502 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:23.601573 5372 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:23.826287 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 102038ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:23.971717 5367 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:23.972200 5367 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:23.974059 5367 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:23.975895 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:00:23.976468 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:00:23.976804 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:23.977438 5367 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:24.060344 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 102038ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:24.110603 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 56: this message will repeat every 5th retry.
I20260501 14:00:24.114048 5378 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
W20260501 14:00:24.115330 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:24.115067 5378 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:24.117084 5378 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:24.118067 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:24.118891 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:24.120141 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:24.120985 5378 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:24.516650 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 102723ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:24.754702 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 102723ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:24.757501 3561 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (101 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 14:00:24.937696 3719 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (71 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 14:00:25.116439 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:25.139382 5384 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:25.140069 5384 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:25.141953 5384 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:25.144774 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:25.144708 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:25.146912 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:25.148087 5384 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:25.222725 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 103418ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:25.244757 5376 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:25.245237 5376 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:25.247193 5376 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:25.251181 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:00:25.252170 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:00:25.252523 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:25.253504 5376 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:25.487727 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 103418ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:25.794973 5376 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:25.795475 5376 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:25.797376 5376 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:25.798815 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:00:25.799615 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:00:25.800001 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:25.800912 5376 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:25.861029 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:25.926626 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 104151ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:26.030365 3867 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (72 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
I20260501 14:00:26.082300 5388 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:26.084702 5388 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:26.096556 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:26.098026 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:26.097429 5388 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:26.099524 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:26.102026 5388 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:26.178951 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 104151ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:26.646040 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 104842ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:26.741496 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:26.887405 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 104842ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x47c8f5 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x49e531 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260501 14:00:27.352893 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 105550ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:27.571069 5396 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:27.571556 5396 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:27.573488 5396 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:27.574930 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:27.576102 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:27.579696 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:27.580374 5396 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:27.597955 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 105550ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:27.769296 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:28.046746 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 106261ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:28.301854 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 106261ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:28.581909 5403 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:28.582643 5403 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:28.584646 5403 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:28.585695 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:28.585681 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:28.588303 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:28.589140 5403 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:28.748976 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 106965ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:28.869884 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:28.976639 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 106965ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:29.431206 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 107639ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:29.571918 5410 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:29.574667 5410 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:29.579964 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:29.584879 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:29.586453 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:29.586192 5410 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:29.587442 5410 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:29.695070 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 107639ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:30.022066 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:30.092105 4046 maintenance_manager.cc:419] P 6b0f8911dd5d431d9d8d3124a4456262: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033399
W20260501 14:00:30.157800 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 108358ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:30.252223 4042 maintenance_manager.cc:643] P 6b0f8911dd5d431d9d8d3124a4456262: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.159s user 0.113s sys 0.016s Metrics: {"bytes_written":36139,"cfile_init":1,"drs_written":1,"lbm_read_time_us":308,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3396,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":94,"thread_start_us":5417,"threads_started":1,"wal-append.queue_time_us":6008}
I20260501 14:00:30.255101 4046 maintenance_manager.cc:419] P 6b0f8911dd5d431d9d8d3124a4456262: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 6381 bytes on disk
I20260501 14:00:30.257184 4042 maintenance_manager.cc:643] P 6b0f8911dd5d431d9d8d3124a4456262: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":311,"lbm_reads_lt_1ms":4}
W20260501 14:00:30.418875 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 108358ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:30.596482 5415 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:30.596995 5415 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:30.598922 5415 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:30.599819 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:30.600382 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:30.602051 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:30.602142 5408 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:30.602741 5408 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:30.602919 5415 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:30.607348 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:00:30.607815 5408 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:30.608345 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:00:30.608739 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:30.609401 5408 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:30.854574 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 109082ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:31.115160 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 109082ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:31.144723 5418 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:31.145205 5418 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:31.147159 5418 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:31.148003 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:31.148569 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:31.149703 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:31.150345 5418 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:31.158504 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:31.559350 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 109778ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:31.728541 5422 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:31.729146 5422 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:31.731631 5422 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:31.731576 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:31.732218 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:31.733726 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:31.734367 5422 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:31.816470 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 109778ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:31.964282 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:32.311228 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 110479ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:32.566511 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 110479ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:33.002163 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 111230ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:33.164216 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:33.249814 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 111230ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:33.698413 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 111913ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:33.935563 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 111913ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:34.083981 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 61: this message will repeat every 5th retry.
W20260501 14:00:34.087482 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:34.097379 5434 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:34.097854 5434 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:34.100392 5434 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:34.100544 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:34.102440 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:34.103767 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:34.104538 5434 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:34.382896 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 112599ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:34.626287 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 112599ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:34.886402 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:35.088440 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 113289ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:35.108023 5442 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:35.108539 5442 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:35.110597 5442 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:35.123719 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:35.124078 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:35.126859 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:35.150985 5442 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:35.336122 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 113289ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:35.668256 5446 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:35.668779 5446 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:35.672333 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:35.672586 5446 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:35.674602 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:35.676178 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:35.677268 5446 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:35.767469 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 113999ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:35.808672 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:36.006572 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 113999ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:36.482486 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 114670ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:36.720646 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 114670ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:36.771663 5453 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:36.772197 5453 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:36.774232 5453 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:36.775085 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:36.775336 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:36.776674 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:36.777710 5453 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:36.808945 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:37.169977 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 115383ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:37.304100 5459 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:37.304690 5459 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:37.319835 5459 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:37.320566 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:37.320781 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:37.322028 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:37.322588 5459 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:37.439096 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 115383ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:37.816519 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:37.872395 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 116102ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:38.125833 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 116102ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:38.265398 5462 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:38.265920 5462 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:38.268702 5462 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:38.268729 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:38.270545 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:38.271941 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:38.272805 5462 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:38.574864 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 116789ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:38.797158 5464 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:38.797626 5464 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:38.799554 5464 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:38.800400 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:38.801489 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:38.802706 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:38.803364 5464 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:38.817744 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 116789ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:38.938585 5461 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:38.939108 5461 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:38.940970 5461 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:38.942271 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:00:38.942958 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:00:38.943266 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:38.943907 5461 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:39.048828 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:39.305781 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 117481ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:39.547881 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 117481ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:39.736410 5472 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:39.736994 5472 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:39.739099 5472 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:39.740244 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:39.741734 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:39.742957 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:39.743571 5472 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:39.919857 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:39.970752 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 118211ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:40.220968 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 118211ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:40.285547 4066 rpcz_store.cc:267] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35438 (request call id 103) took 59999 ms (client timeout 59999 ms). Trace:
W20260501 14:00:40.285965 4066 rpcz_store.cc:269] 0501 13:59:40.285878 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:59:40.285991 (+ 113us) service_pool.cc:224] Handling call
0501 14:00:40.285291 (+59999300us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 14:00:40.289180 3878 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.16.188:38253)
I20260501 14:00:40.295779 3152 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8f8971cc622d4dca8d21a3656e765390"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 14:00:40.301882 3152 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8f8971cc622d4dca8d21a3656e765390 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 14:00:40.333593 3152 catalog_manager.cc:2174] created HMS catalog entry for table [id=5af3134355544f4b9d52e3759232b7ed]
I20260501 14:00:40.334069 5477 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:40.334569 5477 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:40.338006 5477 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:40.343681 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:40.344842 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 225 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:40.346552 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:40.349121 5477 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:00:40.383978 1090 master-stress-test.cc:579] Killed test thread 1/10
I20260501 14:00:40.384286 1090 master-stress-test.cc:577] Killing test thread 2/10
I20260501 14:00:40.384614 1090 master-stress-test.cc:579] Killed test thread 2/10
I20260501 14:00:40.384824 1090 master-stress-test.cc:577] Killing test thread 3/10
I20260501 14:00:40.385174 1090 master-stress-test.cc:579] Killed test thread 3/10
I20260501 14:00:40.385361 1090 master-stress-test.cc:577] Killing test thread 4/10
I20260501 14:00:40.385628 1090 master-stress-test.cc:579] Killed test thread 4/10
I20260501 14:00:40.385828 1090 master-stress-test.cc:577] Killing test thread 5/10
I20260501 14:00:40.386102 1090 master-stress-test.cc:579] Killed test thread 5/10
I20260501 14:00:40.386308 1090 master-stress-test.cc:577] Killing test thread 6/10
I20260501 14:00:40.414098 3651 tablet_service.cc:1511] Processing CreateTablet for tablet c5e5ae7507c74c8990469a556a0c7814 (DEFAULT_TABLE table=default.table_8f8971cc622d4dca8d21a3656e765390 [id=5af3134355544f4b9d52e3759232b7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 14:00:40.415572 3651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5e5ae7507c74c8990469a556a0c7814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 14:00:40.420724 3491 tablet_service.cc:1511] Processing CreateTablet for tablet c5e5ae7507c74c8990469a556a0c7814 (DEFAULT_TABLE table=default.table_8f8971cc622d4dca8d21a3656e765390 [id=5af3134355544f4b9d52e3759232b7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 14:00:40.422091 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5e5ae7507c74c8990469a556a0c7814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 14:00:40.425860 3800 tablet_service.cc:1511] Processing CreateTablet for tablet c5e5ae7507c74c8990469a556a0c7814 (DEFAULT_TABLE table=default.table_8f8971cc622d4dca8d21a3656e765390 [id=5af3134355544f4b9d52e3759232b7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 14:00:40.427274 3800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5e5ae7507c74c8990469a556a0c7814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 14:00:40.470175 5487 tablet_bootstrap.cc:492] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2: Bootstrap starting.
I20260501 14:00:40.475642 5486 tablet_bootstrap.cc:492] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122: Bootstrap starting.
I20260501 14:00:40.475732 5488 tablet_bootstrap.cc:492] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1: Bootstrap starting.
I20260501 14:00:40.483039 5487 tablet_bootstrap.cc:654] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2: Neither blocks nor log segments found. Creating new log.
I20260501 14:00:40.481856 5486 tablet_bootstrap.cc:654] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122: Neither blocks nor log segments found. Creating new log.
I20260501 14:00:40.497625 5486 tablet_bootstrap.cc:492] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122: No bootstrap required, opened a new log
I20260501 14:00:40.498098 5486 ts_tablet_manager.cc:1403] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.010s
I20260501 14:00:40.501092 5486 raft_consensus.cc:359] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 14:00:40.501636 5488 tablet_bootstrap.cc:654] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1: Neither blocks nor log segments found. Creating new log.
I20260501 14:00:40.501792 5486 raft_consensus.cc:385] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 14:00:40.502075 5486 raft_consensus.cc:740] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Initialized, Role: FOLLOWER
I20260501 14:00:40.502760 5486 consensus_queue.cc:260] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 14:00:40.504364 5487 tablet_bootstrap.cc:492] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2: No bootstrap required, opened a new log
I20260501 14:00:40.504438 5486 ts_tablet_manager.cc:1434] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260501 14:00:40.504834 5487 ts_tablet_manager.cc:1403] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.000s
I20260501 14:00:40.508020 5487 raft_consensus.cc:359] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 14:00:40.508935 5487 raft_consensus.cc:385] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 14:00:40.509254 5487 raft_consensus.cc:740] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c239ccbdc554619a237d78e958ffbc2, State: Initialized, Role: FOLLOWER
I20260501 14:00:40.512626 5487 consensus_queue.cc:260] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 14:00:40.515019 5487 ts_tablet_manager.cc:1434] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260501 14:00:40.521385 5488 tablet_bootstrap.cc:492] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1: No bootstrap required, opened a new log
I20260501 14:00:40.521860 5488 ts_tablet_manager.cc:1403] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.000s
I20260501 14:00:40.529853 5488 raft_consensus.cc:359] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 14:00:40.530799 5488 raft_consensus.cc:385] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 14:00:40.531140 5488 raft_consensus.cc:740] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af0cedc7f3c47299b93e6e92d8e02d1, State: Initialized, Role: FOLLOWER
I20260501 14:00:40.531857 5488 consensus_queue.cc:260] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 14:00:40.534163 5488 ts_tablet_manager.cc:1434] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
W20260501 14:00:40.599335 3722 tablet.cc:2404] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 14:00:40.634263 4038 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b9f3b4229594e13959fec0aa8281c03 consensus state has no leader
I20260501 14:00:40.718444 5476 raft_consensus.cc:493] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:40.718902 5476 raft_consensus.cc:515] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 14:00:40.720475 5476 leader_election.cc:290] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 14:00:40.721374 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5e5ae7507c74c8990469a556a0c7814" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" is_pre_election: true
I20260501 14:00:40.721649 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5e5ae7507c74c8990469a556a0c7814" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2" is_pre_election: true
I20260501 14:00:40.721903 3674 raft_consensus.cc:2468] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 14:00:40.726302 3821 raft_consensus.cc:2468] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 0.
I20260501 14:00:40.726511 3448 leader_election.cc:304] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6af0cedc7f3c47299b93e6e92d8e02d1, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 14:00:40.729027 5476 raft_consensus.cc:2804] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 14:00:40.729372 5476 raft_consensus.cc:493] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 14:00:40.729686 5476 raft_consensus.cc:3060] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [term 0 FOLLOWER]: Advancing to term 1
I20260501 14:00:40.738837 5476 raft_consensus.cc:515] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
I20260501 14:00:40.740710 5476 leader_election.cc:290] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Requested vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
I20260501 14:00:40.742484 3674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5e5ae7507c74c8990469a556a0c7814" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1"
I20260501 14:00:40.742573 3821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5e5ae7507c74c8990469a556a0c7814" candidate_uuid: "9154033cea374a10a5225c53b8bcf122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c239ccbdc554619a237d78e958ffbc2"
I20260501 14:00:40.743048 3674 raft_consensus.cc:3060] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 14:00:40.743132 3821 raft_consensus.cc:3060] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2 [term 0 FOLLOWER]: Advancing to term 1
W20260501 14:00:40.745402 3872 tablet.cc:2404] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 14:00:40.750708 3821 raft_consensus.cc:2468] T c5e5ae7507c74c8990469a556a0c7814 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 14:00:40.751821 3674 raft_consensus.cc:2468] T c5e5ae7507c74c8990469a556a0c7814 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9154033cea374a10a5225c53b8bcf122 in term 1.
I20260501 14:00:40.752010 3448 leader_election.cc:304] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c239ccbdc554619a237d78e958ffbc2, 9154033cea374a10a5225c53b8bcf122; no voters:
I20260501 14:00:40.753212 5475 raft_consensus.cc:2804] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 14:00:40.753697 5475 raft_consensus.cc:697] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [term 1 LEADER]: Becoming Leader. State: Replica: 9154033cea374a10a5225c53b8bcf122, State: Running, Role: LEADER
I20260501 14:00:40.754539 5475 consensus_queue.cc:237] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } }
W20260501 14:00:40.758083 3568 tablet.cc:2404] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 14:00:40.765858 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 118884ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:40.769690 3154 catalog_manager.cc:5671] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9154033cea374a10a5225c53b8bcf122 (127.1.16.129). New cstate: current_term: 1 leader_uuid: "9154033cea374a10a5225c53b8bcf122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } health_report { overall_health: HEALTHY } } }
I20260501 14:00:40.866039 5475 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:40.866564 5475 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:40.868570 5475 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:40.871181 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:00:40.872625 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:00:40.872977 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:40.873651 5475 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:40.894619 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:41.011559 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 118884ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:41.196249 4064 rpcz_store.cc:267] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35438 (request call id 107) took 59999 ms (client timeout 59999 ms). Trace:
W20260501 14:00:41.196552 4063 rpcz_store.cc:267] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35438 (request call id 108) took 59999 ms (client timeout 59999 ms). Trace:
W20260501 14:00:41.196645 4064 rpcz_store.cc:269] 0501 13:59:41.196684 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:59:41.198941 (+ 2257us) service_pool.cc:224] Handling call
0501 14:00:41.195995 (+59997054us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 14:00:41.196981 4063 rpcz_store.cc:269] 0501 13:59:41.197003 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 13:59:41.197131 (+ 128us) service_pool.cc:224] Handling call
0501 14:00:41.196336 (+59999205us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 14:00:41.199267 3878 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.16.190:38561)
W20260501 14:00:41.200133 3878 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.16.190:38561)
I20260501 14:00:41.205119 4063 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_0da08f161093429e83b826825c3a9727" } modify_external_catalogs: true
I20260501 14:00:41.205456 4064 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_124cdf87baa148fbbc6e5f06d1efcda1" } new_table_name: "default.Table_83691904199e4bcaa0dd9732f655dec6" modify_external_catalogs: true
I20260501 14:00:41.206574 4063 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35438:
table { table_name: "default.Table_0da08f161093429e83b826825c3a9727" } modify_external_catalogs: true
W20260501 14:00:41.443524 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 119675ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:41.523415 3878 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.16.188:38253)
I20260501 14:00:41.536331 4066 master_service.cc:954] ReplaceTablet: received request to replace tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 from {username='slave'} at 127.0.0.1:35438
W20260501 14:00:41.668818 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 119675ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:41.797540 5476 consensus_queue.cc:1048] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 14:00:41.833822 5505 consensus_queue.cc:1048] T c5e5ae7507c74c8990469a556a0c7814 P 9154033cea374a10a5225c53b8bcf122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 14:00:41.993480 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:42.119917 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 120332ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:42.361820 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 120332ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:42.751914 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:42.763656 5513 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:42.764686 5513 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:42.766640 5513 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:42.767381 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:42.767518 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:42.768959 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:42.769641 5513 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:42.808531 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 121025ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:42.928804 5509 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:42.929287 5509 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:42.931218 5509 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:42.934528 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:00:42.935132 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:00:42.935454 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:42.936506 5509 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:43.048161 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 121025ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:43.265645 4523 maintenance_manager.cc:419] P 15dd70d7c5494c3c911f0a780dc9a2af: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033361
I20260501 14:00:43.379171 5515 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:43.379791 5515 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:43.382582 5515 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:43.382751 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:43.383277 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:43.384548 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:43.386023 5515 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:00:43.435806 4521 maintenance_manager.cc:643] P 15dd70d7c5494c3c911f0a780dc9a2af: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.169s user 0.098s sys 0.034s Metrics: {"bytes_written":38491,"cfile_init":1,"drs_written":1,"lbm_read_time_us":325,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6492,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":100,"thread_start_us":7150,"threads_started":1,"wal-append.queue_time_us":6900}
I20260501 14:00:43.438227 4523 maintenance_manager.cc:419] P 15dd70d7c5494c3c911f0a780dc9a2af: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7355 bytes on disk
I20260501 14:00:43.441151 4521 maintenance_manager.cc:643] P 15dd70d7c5494c3c911f0a780dc9a2af: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":289,"lbm_reads_lt_1ms":4}
W20260501 14:00:43.504703 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 121711ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:43.602195 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 66: this message will repeat every 5th retry.
W20260501 14:00:43.613072 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:43.776237 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 121711ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:43.929738 5521 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:43.930125 5521 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:43.932013 5521 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:43.932765 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:43.932957 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:43.934280 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:43.935221 5521 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:44.225386 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 122439ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:44.424147 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:44.481932 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 122439ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:44.583243 5526 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:44.583803 5526 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:44.585786 5526 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:44.586565 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:44.586931 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:44.590583 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:44.591405 5526 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:44.617081 3715 debug-util.cc:398] Leaking SignalData structure 0x7b08000de1e0 after lost signal to thread 3572
W20260501 14:00:44.618260 3715 debug-util.cc:398] Leaking SignalData structure 0x7b08000cf960 after lost signal to thread 3718
W20260501 14:00:44.912197 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 123145ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:45.181759 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 123145ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:45.421803 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:45.657570 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 123845ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:45.895674 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 123845ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:45.983731 5531 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:45.984441 5531 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:45.986676 5531 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:45.987715 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:45.988348 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:45.990141 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:45.990769 5531 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:46.353871 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 124559ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:46.529435 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:46.609041 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 124559ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:47.048874 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 125272ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:47.091411 5536 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:47.092128 5536 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:47.094951 5536 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:47.097496 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:00:47.098170 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:00:47.098516 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:47.099917 5536 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:47.303961 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 125272ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:47.454753 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:47.754899 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 125967ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:47.987877 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 125967ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:48.349831 5540 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:48.350332 5540 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:48.352365 5540 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:48.353379 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:48.353863 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:48.355507 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:48.356716 5540 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:48.422772 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 126651ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:48.475109 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:48.668286 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 126651ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:49.124740 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 127331ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:49.288865 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:49.375455 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 127331ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:49.823936 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 128038ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:50.073053 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 128038ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:50.375000 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:50.516904 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 128736ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:50.765085 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 128736ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:50.985200 5552 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:50.985809 5552 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:50.987856 5552 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:50.988745 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:50.988722 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:50.990204 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:50.990864 5552 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:51.229758 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 129428ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:51.414346 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:51.484529 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 129428ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:51.631047 5550 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:51.631438 5550 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:51.633246 5550 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:51.634711 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:00:51.635226 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:00:51.635516 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:51.636011 5550 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:51.927067 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 130147ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:51.999490 5558 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:52.000224 5558 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:52.003741 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:52.004371 5558 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:52.004554 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:52.005913 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:52.018638 5558 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:52.183657 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 130147ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:52.359201 5561 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:00:52.359704 5561 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:00:52.361599 5561 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:00:52.363183 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:00:52.363819 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:00:52.364151 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:00:52.364801 5561 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:00:52.533557 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:00:52.621049 5565 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:52.621546 5565 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:52.623413 5565 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:52.624951 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:52.626312 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:52.630079 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:52.632543 5565 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:52.645256 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 130846ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x47c8f5 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x49e531 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260501 14:00:52.889467 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 130846ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:53.177999 5567 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:53.178418 5567 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:53.179971 5567 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:53.180716 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:53.180884 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:53.182173 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:53.182866 5567 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:53.324838 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 131552ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:53.561970 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 131552ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:53.718216 5569 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:53.718745 5569 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:53.720525 5569 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:53.721203 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:53.721436 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:53.723002 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:53.723580 5569 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:53.774330 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 71: this message will repeat every 5th retry.
W20260501 14:00:53.777908 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:54.009440 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 132225ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:54.246564 5574 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:54.248718 5574 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:54.251670 5574 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:54.252602 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:54.252602 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:54.254218 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:54.255219 5574 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:54.260852 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 132225ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:54.708011 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 132924ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:54.772639 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:54.950212 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 132924ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:55.396557 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 133614ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:55.646636 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 133614ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:55.993482 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:56.078891 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 134309ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:56.103492 5583 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:56.104542 5583 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:56.108443 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:56.112177 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:56.116705 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:56.110881 5583 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:56.123642 5583 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:56.317101 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 134309ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:56.683329 5587 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:56.683920 5587 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:56.688102 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:56.689548 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:56.692452 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:56.699374 5587 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:56.700459 5587 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:00:56.775027 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 134980ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:57.015542 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 134980ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:57.025465 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:57.440794 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 135679ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:57.684279 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 135679ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:57.707515 3557 debug-util.cc:398] Leaking SignalData structure 0x7b08000f1160 after lost signal to thread 3422
W20260501 14:00:57.708794 3557 debug-util.cc:398] Leaking SignalData structure 0x7b08000d6600 after lost signal to thread 3560
W20260501 14:00:57.891274 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:58.123117 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 136349ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:58.182498 3863 debug-util.cc:398] Leaking SignalData structure 0x7b080010a5a0 after lost signal to thread 3730
W20260501 14:00:58.183670 3863 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea0e0 after lost signal to thread 3866
W20260501 14:00:58.372915 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 136349ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:58.814994 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 137036ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:58.945397 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:00:59.059784 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 137036ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:59.493788 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 137723ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:00:59.741395 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 137723ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:00:59.802090 5601 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:00:59.802634 5601 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:00:59.804560 5601 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:00:59.805756 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:00:59.805855 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:00:59.807328 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:00:59.808010 5601 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:00.200290 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 138405ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:00.205982 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:00.465878 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 138405ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:00.749480 5605 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:00.749949 5605 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:00.752063 5605 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:00.753028 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:00.753552 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:00.755043 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:00.755728 5605 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:00.926653 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 139129ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:01.180621 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 139129ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:01.289549 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:01.628052 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 139844ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:01.886572 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 139844ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:02.067198 5611 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:02.067699 5611 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:02.069986 5611 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:02.070731 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:02.071038 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:02.072602 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:02.074110 5611 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:02.349745 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 140550ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:02.412675 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:02.602670 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 140550ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:02.654320 5616 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:02.654930 5616 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:02.656879 5616 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:02.657873 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:02.658272 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:02.667603 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:02.669159 5616 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:03.038722 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 141266ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:03.216997 5619 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:03.218076 5619 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:03.222043 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:03.224603 5619 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:03.227567 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:03.228914 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:03.229671 5619 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:03.289316 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 141265ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:03.409605 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:01:03.747526 5625 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:03.750142 5625 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:03.752081 5625 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:03.752769 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:03.752993 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:03.754792 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:03.756497 5625 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:03.752986 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 141952ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:03.996867 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 141952ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:04.337299 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 76: this message will repeat every 5th retry.
W20260501 14:01:04.340451 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:04.453002 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 142660ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:04.702998 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 142660ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:05.150924 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 143366ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:05.223199 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:05.398366 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 143366ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:05.848862 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 144061ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:06.038259 5632 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:06.038867 5632 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:06.041163 5632 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:06.044658 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:01:06.045323 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:01:06.045662 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:06.046367 5632 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:06.104818 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 144061ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:06.166407 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:01:06.277251 5637 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:06.277767 5637 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:06.279600 5637 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:06.280323 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:06.281149 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:06.282900 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:06.283455 5637 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:06.538376 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 144768ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:06.795181 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 144768ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:06.992399 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:07.244897 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 145458ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:07.495273 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 145458ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:07.941540 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 146159ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:08.116817 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:08.172123 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 146159ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:08.610724 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 146835ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:08.630136 5648 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:08.630856 5648 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:08.634268 5648 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:08.638073 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:08.638255 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:08.639909 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:08.640877 5648 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:08.860251 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 146835ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:09.269794 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:09.298899 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 147523ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:09.540937 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 147523ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:09.676915 5653 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:09.677549 5653 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:09.679747 5653 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:09.682045 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:01:09.682744 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:01:09.683076 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:09.683878 5653 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:09.995941 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 148204ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:10.102183 5657 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:10.102697 5657 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:10.104635 5657 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:10.106037 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:10.106329 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:10.109665 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:10.110212 5657 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:10.168093 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:10.231906 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 148204ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:10.678141 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 148895ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:10.705973 5660 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:10.706595 5660 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:10.709299 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:10.711997 5660 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:10.712082 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:10.713346 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:10.714350 5660 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:10.924875 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 148895ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:11.051460 5658 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:11.051977 5658 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:11.053948 5658 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:11.056289 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:01:11.056927 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:01:11.057245 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:11.057898 5658 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:11.236788 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:11.374150 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 149588ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:11.616829 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 149588ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:12.031661 5665 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:12.032238 5665 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:12.034154 5665 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:12.035712 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:12.035702 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:12.038604 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:12.039738 5665 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:12.074204 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 150280ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:12.252679 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:12.329627 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 150280ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:12.671350 5671 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:12.671941 5671 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:12.674696 5671 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:12.675691 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:12.676101 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:12.677349 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:12.678318 5671 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:12.787557 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 150993ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:13.038959 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 150993ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:13.266353 5673 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:13.267072 5673 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:13.270287 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:13.271299 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:13.271942 5673 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:13.275485 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:13.278510 5673 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:13.345547 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:13.489199 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 151702ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:13.737737 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 151702ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:14.190191 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 152401ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:14.427987 3139 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.1.16.188:38253 timed out after 30.000s (SENT)
W20260501 14:01:14.438773 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 152401ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:14.585916 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 81: this message will repeat every 5th retry.
W20260501 14:01:14.588042 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:01:14.709615 5682 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:14.710213 5682 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:14.712606 5682 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:14.714062 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:14.715348 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:14.716643 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:14.717545 5682 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:14.895771 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 153102ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:15.174827 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 153102ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:15.289609 5687 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:15.290524 5687 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:15.292948 5687 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:15.293569 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:15.294396 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:15.296339 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:15.297223 5687 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:15.626797 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 153838ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:15.761137 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:15.888427 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 153838ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:16.033617 3715 debug-util.cc:398] Leaking SignalData structure 0x7b08000adf80 after lost signal to thread 3572
W20260501 14:01:16.034808 3715 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0ec0 after lost signal to thread 3718
W20260501 14:01:16.349359 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 154551ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:16.592633 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 154551ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:16.647657 5692 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:16.648152 5692 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:16.650068 5692 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:16.650780 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:16.650976 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:16.652149 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:16.652863 5692 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:16.713889 4110 debug-util.cc:398] Leaking SignalData structure 0x7b0800075ca0 after lost signal to thread 3931
W20260501 14:01:16.714490 4110 debug-util.cc:398] Leaking SignalData structure 0x7b080007f1a0 after lost signal to thread 4114
W20260501 14:01:16.990398 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:17.031692 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 155256ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:17.287213 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 155256ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:17.654343 5696 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:17.654902 5696 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:17.656824 5696 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:17.657773 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:17.657802 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:17.659273 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:17.659950 5696 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:01:17.704849 5695 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:17.705497 5695 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:17.707633 5695 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:17.709048 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:01:17.709676 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:01:17.709961 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:17.710511 5695 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:17.735086 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 155950ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:17.770313 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:17.984021 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 155950ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:18.435119 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 156647ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:18.688629 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:18.694183 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 156647ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:18.835086 5699 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:18.835574 5699 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:18.837466 5699 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:18.839169 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:01:18.839728 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:01:18.840070 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:18.840685 5699 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:19.153570 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 157357ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:19.405738 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 157357ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:19.499153 5703 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:19.499650 5703 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:19.501382 5703 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:19.502509 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:19.503217 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:19.504688 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:19.505357 5703 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:19.838161 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:19.891801 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 158069ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x47c8f5 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x4959cf __interceptor_pthread_setcancelstate at ??:?
@ 0x7f493f7d8fc0 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260501 14:01:20.147035 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 158069ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:20.590693 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 158810ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:20.761600 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:20.840416 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 158810ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:20.978654 5713 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:20.979053 5713 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:20.980665 5713 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:20.981654 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:20.981662 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:20.983378 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:20.984398 5713 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:21.303264 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 159503ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:21.502130 5715 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:21.502683 5715 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:21.504565 5715 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:21.506428 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:21.507530 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:21.510308 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:21.511008 5715 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:21.540158 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 159503ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:21.958323 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:22.000087 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 160203ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:22.078801 5720 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:22.079350 5720 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:22.081288 5720 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:22.082736 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:22.084263 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:22.087028 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:22.087771 5720 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:22.276115 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 160203ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:22.710657 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 160939ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:22.963411 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 160939ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:23.163879 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:23.410918 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 161626ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:23.582484 5728 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:23.583021 5728 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:23.584959 5728 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:23.586265 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:23.588307 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:23.589804 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:23.591076 5728 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:23.651119 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 161626ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:23.981648 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:24.097554 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 162314ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:24.130187 5734 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:24.130860 5734 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:24.133399 5734 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:24.134356 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:24.134619 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:24.135872 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:24.136415 5734 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:24.355533 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 162314ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:24.801505 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 163018ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:25.051788 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 163018ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:25.166870 3561 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (160 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 14:01:25.202294 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 86: this message will repeat every 5th retry.
W20260501 14:01:25.208837 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:25.496320 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 163715ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:25.538885 3719 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (112 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
I20260501 14:01:25.614001 5738 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:25.614548 5738 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:25.616979 5738 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:25.620705 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:01:25.621551 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:01:25.621892 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:25.622669 5738 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:25.735386 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 163715ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:25.969990 5741 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:25.970894 5741 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:25.973352 5741 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:25.974155 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:25.975167 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:25.977360 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:25.978219 5741 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:26.191773 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 164399ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:26.294910 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:26.441349 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 164399ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:26.516567 5748 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:26.517073 5748 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:26.519224 5748 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:26.519829 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:26.520114 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:26.521239 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:26.521888 5748 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:26.628347 3867 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (113 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 14:01:26.888216 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 165104ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:27.047271 5751 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:27.047778 5751 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:27.049763 5751 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:27.050348 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:27.050681 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:27.052359 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:27.075579 5751 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:27.149220 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 165104ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:27.329700 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:27.594336 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 165812ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:27.841548 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 165812ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:28.130074 5757 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:28.130648 5757 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:28.134866 5757 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:28.134829 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:28.135478 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:28.136780 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:28.137697 5757 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:28.299351 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 166505ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:28.539981 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:28.544988 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 166505ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:29.011495 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 167208ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:29.248390 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 167208ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:29.302191 5759 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:29.302772 5759 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:29.304526 5759 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:29.306046 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:01:29.306639 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:01:29.306890 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:29.307426 5759 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:29.391580 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:29.685063 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 167911ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:29.926297 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 167911ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:30.004534 5767 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:30.005249 5767 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:30.007824 5767 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:30.008721 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:30.009027 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:30.010664 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:30.011334 5767 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:30.378722 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 168589ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:30.461582 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:01:30.605082 5773 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:30.605966 5773 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:30.608421 5773 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:30.609694 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:30.609803 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:30.611250 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:30.611913 5773 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:30.624132 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 168589ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:31.056135 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 169287ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:31.316259 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 169287ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:31.444140 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:31.754823 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 169979ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:31.991767 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 169979ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:32.367259 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:32.424609 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 170655ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:32.509248 5781 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:32.509807 5781 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:32.511719 5781 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:32.513042 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:32.514355 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:32.515610 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:32.516180 5781 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:32.670717 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 170655ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:33.046738 5785 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:33.047971 5785 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:33.056309 5785 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:33.056586 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:33.058579 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:33.059703 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:33.067095 5785 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:33.118220 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 171334ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:33.149020 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:33.368240 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 171334ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:33.804807 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 172031ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:34.031461 5791 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:34.031972 5791 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:34.035004 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:34.038822 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:34.040365 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:34.046284 5791 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:34.047330 5792 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:34.079049 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 172031ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:34.266203 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:34.521381 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 172742ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:34.760322 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 172742ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:35.099068 5799 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:35.099551 5799 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:35.101485 5799 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:35.102620 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:35.104759 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:35.106456 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:35.108933 5799 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:35.223730 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 173423ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:35.228526 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:35.230446 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 91: this message will repeat every 5th retry.
I20260501 14:01:35.274868 5795 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:35.275393 5795 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:35.277300 5795 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:35.280378 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:01:35.280953 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:01:35.281271 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:35.281937 5795 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 14:01:35.443938 5795 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:35.444334 5795 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:35.446079 5795 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:35.447934 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:01:35.448522 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:01:35.448868 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:35.449514 5795 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:35.461265 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 173423ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:35.664822 5804 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:35.665546 5804 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:35.667968 5804 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:35.669030 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:35.669612 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:35.672334 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:35.673146 5804 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:35.977310 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 174124ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:36.096082 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:36.221553 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 174124ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:36.661643 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 174884ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x47c8f5 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x4959cf __interceptor_pthread_setcancelstate at ??:?
@ 0x7f493f7d8fc0 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260501 14:01:36.903802 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 174884ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:37.131574 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:37.358006 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 175567ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:37.505615 5813 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:37.506136 5813 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:37.508309 5813 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:37.508867 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:37.510885 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:37.512188 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:37.513646 5813 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:37.588351 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 175567ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:38.019827 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 176251ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:38.164314 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:38.279191 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 176251ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:38.544235 5816 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:38.544728 5816 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:38.546569 5816 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:38.547991 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:38.548199 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:38.550031 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:38.550711 5816 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:38.716539 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 176942ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:38.957482 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 176942ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:39.007233 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:01:39.059938 5821 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:39.060447 5821 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:39.062520 5821 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:39.063504 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:39.064087 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:39.065529 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:39.066183 5821 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:39.406224 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 177620ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:39.659114 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 177620ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:39.801772 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:40.133817 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 178322ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:40.387010 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 178322ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:40.546252 5828 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:40.546693 5828 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:40.548898 5828 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:40.549579 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:40.549760 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:40.551245 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:40.551749 5828 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:40.667297 4038 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b9f3b4229594e13959fec0aa8281c03 consensus state has no leader
W20260501 14:01:40.834151 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 179050ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:41.047937 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:41.093595 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 179050ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:41.150482 5832 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:41.151001 5832 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:41.153537 5832 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:41.153486 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:41.154127 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 229 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:41.155525 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:41.156185 5832 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:41.204275 4063 rpcz_store.cc:267] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35438 (request call id 113) took 59999 ms (client timeout 59999 ms). Trace:
W20260501 14:01:41.204548 4064 rpcz_store.cc:267] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35438 (request call id 114) took 59999 ms (client timeout 59999 ms). Trace:
W20260501 14:01:41.204829 4064 rpcz_store.cc:269] 0501 14:00:41.204969 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 14:00:41.205087 (+ 118us) service_pool.cc:224] Handling call
0501 14:01:41.204315 (+59999228us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 14:01:41.204772 4063 rpcz_store.cc:269] 0501 14:00:41.204619 (+ 0us) service_pool.cc:167] Inserting onto call queue
0501 14:00:41.204754 (+ 135us) service_pool.cc:224] Handling call
0501 14:01:41.203998 (+59999244us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260501 14:01:41.208483 3878 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.16.188:38253)
W20260501 14:01:41.209407 3878 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.16.188:38253)
I20260501 14:01:41.214592 3152 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0da08f161093429e83b826825c3a9727" } modify_external_catalogs: true
I20260501 14:01:41.215054 3154 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_124cdf87baa148fbbc6e5f06d1efcda1" } new_table_name: "default.Table_83691904199e4bcaa0dd9732f655dec6" modify_external_catalogs: true
I20260501 14:01:41.215423 3152 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0da08f161093429e83b826825c3a9727" } modify_external_catalogs: true
I20260501 14:01:41.260576 3152 catalog_manager.cc:2731] deleted HMS catalog entry for table default.table_0da08f161093429e83b826825c3a9727 [id=5b3f55db8abb4fa7a9b3abf286620942]
I20260501 14:01:41.349692 3154 catalog_manager.cc:3502] renamed table default.table_124cdf87baa148fbbc6e5f06d1efcda1 [id=4f022a0cee6540b5959fd48b029df1af] in HMS: new name default.table_83691904199e4bcaa0dd9732f655dec6
I20260501 14:01:41.356285 3417 catalog_manager.cc:2772] Deleting table default.table_0da08f161093429e83b826825c3a9727 [id=5b3f55db8abb4fa7a9b3abf286620942] in response to Hive Metastore notification log event 37
I20260501 14:01:41.435299 3417 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: Sending DeleteTablet for 3 replicas of tablet 151c6fb3afbd46a9a4c9cac128ad4ec0
I20260501 14:01:41.439906 3651 tablet_service.cc:1558] Processing DeleteTablet for tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 14:01:41 UTC) from {username='slave'} at 127.0.0.1:35134
I20260501 14:01:41.440097 3800 tablet_service.cc:1558] Processing DeleteTablet for tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 14:01:41 UTC) from {username='slave'} at 127.0.0.1:46442
I20260501 14:01:41.441716 5841 tablet_replica.cc:333] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1: stopping tablet replica
I20260501 14:01:41.441891 3491 tablet_service.cc:1558] Processing DeleteTablet for tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 14:01:41 UTC) from {username='slave'} at 127.0.0.1:38966
I20260501 14:01:41.442636 5841 raft_consensus.cc:2243] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 14:01:41.443781 5841 raft_consensus.cc:2272] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 14:01:41.449334 5841 ts_tablet_manager.cc:1916] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 14:01:41.454938 5843 tablet_replica.cc:333] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122: stopping tablet replica
I20260501 14:01:41.460106 5843 raft_consensus.cc:2243] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 14:01:41.461001 5843 raft_consensus.cc:2272] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 14:01:41.463371 5843 ts_tablet_manager.cc:1916] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 14:01:41.468469 5842 tablet_replica.cc:333] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2: stopping tablet replica
I20260501 14:01:41.470408 5841 ts_tablet_manager.cc:1929] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 14:01:41.470950 5841 log.cc:1199] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-1/wal/wals/151c6fb3afbd46a9a4c9cac128ad4ec0
I20260501 14:01:41.471947 5841 ts_tablet_manager.cc:1950] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 6af0cedc7f3c47299b93e6e92d8e02d1: Deleting consensus metadata
I20260501 14:01:41.478765 3139 catalog_manager.cc:5002] TS 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 (table default.table_0da08f161093429e83b826825c3a9727 [id=5b3f55db8abb4fa7a9b3abf286620942]) successfully deleted
I20260501 14:01:41.482723 5842 raft_consensus.cc:2243] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 14:01:41.483949 5842 raft_consensus.cc:2272] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 14:01:41.486232 5842 ts_tablet_manager.cc:1916] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 14:01:41.498138 5843 ts_tablet_manager.cc:1929] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 14:01:41.503732 5842 ts_tablet_manager.cc:1929] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260501 14:01:41.504117 5842 log.cc:1199] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-2/wal/wals/151c6fb3afbd46a9a4c9cac128ad4ec0
I20260501 14:01:41.505221 5842 ts_tablet_manager.cc:1950] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 5c239ccbdc554619a237d78e958ffbc2: Deleting consensus metadata
I20260501 14:01:41.498695 5843 log.cc:1199] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122: Deleting WAL directory at /tmp/dist-test-task2LI5eU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777643832884646-1090-0/minicluster-data/ts-0/wal/wals/151c6fb3afbd46a9a4c9cac128ad4ec0
I20260501 14:01:41.514540 5843 ts_tablet_manager.cc:1950] T 151c6fb3afbd46a9a4c9cac128ad4ec0 P 9154033cea374a10a5225c53b8bcf122: Deleting consensus metadata
I20260501 14:01:41.515914 1090 master-stress-test.cc:579] Killed test thread 6/10
I20260501 14:01:41.516182 1090 master-stress-test.cc:577] Killing test thread 7/10
I20260501 14:01:41.516945 1090 master-stress-test.cc:579] Killed test thread 7/10
I20260501 14:01:41.517153 1090 master-stress-test.cc:577] Killing test thread 8/10
I20260501 14:01:41.516907 3139 catalog_manager.cc:5002] TS 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 (table default.table_0da08f161093429e83b826825c3a9727 [id=5b3f55db8abb4fa7a9b3abf286620942]) successfully deleted
I20260501 14:01:41.517422 1090 master-stress-test.cc:579] Killed test thread 8/10
I20260501 14:01:41.517612 1090 master-stress-test.cc:577] Killing test thread 9/10
I20260501 14:01:41.518009 3140 catalog_manager.cc:5002] TS 9154033cea374a10a5225c53b8bcf122 (127.1.16.129:44335): tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 (table default.table_0da08f161093429e83b826825c3a9727 [id=5b3f55db8abb4fa7a9b3abf286620942]) successfully deleted
W20260501 14:01:41.537676 3878 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.16.190:38561)
I20260501 14:01:41.555929 5851 tablet.cc:1724] T 6087f52d19984dc2a83b6f923685610a P 5c239ccbdc554619a237d78e958ffbc2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 14:01:41.563429 4063 master_service.cc:954] ReplaceTablet: received request to replace tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 from {username='slave'} at 127.0.0.1:35438
I20260501 14:01:41.563613 5850 tablet.cc:1724] T 6087f52d19984dc2a83b6f923685610a P 6af0cedc7f3c47299b93e6e92d8e02d1: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260501 14:01:41.584838 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 179757ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:41.589546 5853 tablet.cc:1724] T 6087f52d19984dc2a83b6f923685610a P 9154033cea374a10a5225c53b8bcf122: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260501 14:01:41.605852 3154 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 0a1d2cd221924677a3ef81b081e5e016: default.table_83691904199e4bcaa0dd9732f655dec6 [id=4f022a0cee6540b5959fd48b029df1af] alter complete (version 1)
W20260501 14:01:41.834821 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 179757ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:42.274191 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:42.306906 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 180498ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:42.552714 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 180498ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:43.001338 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 181216ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:43.041210 5859 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:43.041890 5859 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:43.044844 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:43.048166 5859 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:43.049005 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:43.050402 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:43.058672 5859 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:01:43.082324 5858 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:43.083160 5858 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:43.085470 5858 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:43.089111 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:01:43.090349 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:01:43.090708 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:43.091327 5858 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:43.220477 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:43.280450 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 181216ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:43.705946 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 181943ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:43.962502 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 181943ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:44.071676 5863 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:44.072247 5863 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:44.074903 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:44.075374 5863 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:44.076074 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:44.077533 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:44.078212 5863 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:44.304273 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:44.398348 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 182625ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:44.592670 5869 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:44.593184 5869 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:44.595125 5869 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:44.596119 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:44.596061 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:44.597776 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:44.598280 5869 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:44.629607 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 182625ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:45.066268 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 183293ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:45.192153 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 96: this message will repeat every 5th retry.
W20260501 14:01:45.197021 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:01:45.206096 5873 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:45.207026 5873 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:45.210803 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:45.212157 5873 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:45.212311 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:45.214619 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:45.215333 5873 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:45.327420 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 183293ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:45.760295 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 183991ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:45.979182 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:45.997711 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 183991ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:46.397158 5872 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:46.397665 5872 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:46.399643 5872 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:46.401924 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:01:46.402626 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:01:46.402990 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:46.403640 5872 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:46.454732 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 184661ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:46.691749 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 184661ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:46.693552 5881 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:46.694213 5881 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:46.696691 5881 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:46.697363 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:46.697770 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:46.699554 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:46.700325 5881 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:46.789399 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:47.140609 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 185355ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:47.378680 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 185355ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:47.722419 5888 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:47.722963 5888 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:47.724953 5888 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:47.726413 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:47.727697 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:47.729585 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:47.730423 5888 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:47.808581 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 186041ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:47.866413 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:48.050091 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 186041ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:48.491533 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 186713ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:48.695510 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:48.736964 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 186713ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:49.210953 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 187400ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:49.219823 5896 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:49.220341 5896 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:49.222334 5896 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:49.227190 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:49.227294 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:49.231237 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:49.231985 5896 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:49.465619 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 187400ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:49.545095 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:49.901455 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 188129ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:50.150561 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 188129ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x47c8f5 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x49e531 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260501 14:01:50.398810 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:50.590559 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 188813ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:50.834249 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 188813ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:51.325927 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 189497ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:51.552910 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 189497ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:51.626065 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:01:51.663414 5905 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:51.663790 5905 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:51.666425 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:51.666676 5905 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:51.667953 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:51.669164 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:51.669799 5905 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:52.018733 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 190216ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:52.271572 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 190216ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:52.275461 5907 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:52.275991 5907 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:52.278198 5907 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:52.278893 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:52.279268 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:52.280655 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:52.281339 5907 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:52.495563 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:52.615504 3185 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9020 after lost signal to thread 3122
W20260501 14:01:52.616226 3185 debug-util.cc:398] Leaking SignalData structure 0x7b080009a620 after lost signal to thread 3188
W20260501 14:01:52.710408 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 190934ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:52.898052 5912 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:52.898576 5912 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:52.900424 5912 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:52.901506 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:52.901907 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:52.903126 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:52.903635 5912 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:52.964569 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 190934ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:53.402845 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:53.418121 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 191628ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:53.563077 5914 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:53.563632 5914 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:53.565529 5914 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:53.567315 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:01:53.567857 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:01:53.568174 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:53.568959 5914 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:53.660070 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 191628ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:54.102849 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 192323ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:54.327520 5919 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:54.328035 5919 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:54.330039 5919 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:54.331473 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:54.332676 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:54.333894 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:54.334650 5919 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:54.347189 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 192323ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:54.379420 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 101: this message will repeat every 5th retry.
W20260501 14:01:54.383675 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:01:54.637094 5920 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:01:54.637568 5920 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:01:54.640213 5920 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:01:54.641855 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:01:54.642510 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:01:54.642840 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:01:54.643468 5920 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:01:54.795789 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 193010ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:55.025444 3863 debug-util.cc:398] Leaking SignalData structure 0x7b080003fb00 after lost signal to thread 3730
W20260501 14:01:55.026690 3863 debug-util.cc:398] Leaking SignalData structure 0x7b08000f0740 after lost signal to thread 3866
W20260501 14:01:55.033806 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 193010ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:55.290455 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:55.482792 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 193697ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:55.714912 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 193697ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:56.120399 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:56.167870 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 194378ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:56.395483 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 194378ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:56.771982 5933 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:56.772672 5933 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:56.775656 5933 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:56.778918 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:56.779091 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:56.782094 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:56.783846 5933 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:56.869767 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 195058ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:57.123587 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 195058ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:57.360124 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:57.445488 4617 debug-util.cc:398] Leaking SignalData structure 0x7b080009d3e0 after lost signal to thread 4319
W20260501 14:01:57.446179 4617 debug-util.cc:398] Leaking SignalData structure 0x7b08000749e0 after lost signal to thread 4620
W20260501 14:01:57.561606 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 195786ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:57.809180 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 195786ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:58.219188 5940 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:58.219770 5940 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:58.221906 5940 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:58.222621 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:58.222929 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:58.224418 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:58.225075 5940 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:58.256276 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 196472ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:58.324885 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:58.496330 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 196472ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:58.768766 5945 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:58.769336 5945 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:58.772707 5945 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:58.773558 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:58.774031 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:58.776495 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:58.777921 5945 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:58.944628 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 197159ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:59.209560 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 197159ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:01:59.294097 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:01:59.644328 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 197872ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:01:59.700335 5950 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:01:59.700879 5950 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:01:59.702759 5950 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:01:59.703696 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:01:59.703961 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:01:59.705569 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:01:59.706351 5950 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:01:59.905098 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 197872ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:00.319183 5953 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:00.319705 5953 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:00.321329 5953 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:00.322786 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:00.322979 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:00.324604 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:00.325372 5953 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:00.344380 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 198568ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:00.371186 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:00.573174 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 198568ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:00.920058 5957 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:00.920676 5957 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:00.922804 5957 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:00.923314 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:00.924983 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:00.926199 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:00.926810 5957 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:01.022337 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 199236ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:01.273327 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 199236ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:01.411254 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:01.720702 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 199936ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:01.971031 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 199936ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:02.314163 5960 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:02.314824 5960 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:02.317566 5960 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:02.320149 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:02:02.320763 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:02:02.321096 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:02.321758 5960 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:02.406584 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 200634ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:02.545804 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:02.648344 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 200634ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:02.895263 5966 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:02.895828 5966 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:02.897782 5966 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:02.898661 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:02.898860 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:02.900254 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:02.900852 5966 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:03.090829 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 201311ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:03.341593 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 201311ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:03.411401 5968 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:03.411913 5968 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:03.413843 5968 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:03.414906 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:03.417537 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:03.418766 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:03.419462 5968 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:03.629771 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:03.802670 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 202004ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:03.934801 5974 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:03.935323 5974 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:03.937373 5974 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:03.938104 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:03.938369 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:03.939723 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:03.940227 5974 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:04.052050 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 202004ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:04.489023 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 202715ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:04.614953 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 106: this message will repeat every 5th retry.
W20260501 14:02:04.619648 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:04.702157 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 202715ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:05.155063 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 203365ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:05.434341 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 203365ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:05.589186 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:05.820428 5986 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:05.821169 5986 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:05.823403 5986 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:05.824651 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:05.825757 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:05.827756 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:05.829082 5986 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:05.884956 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 204098ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:06.137497 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 204098ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:06.293377 3715 debug-util.cc:398] Leaking SignalData structure 0x7b08000be5c0 after lost signal to thread 3572
W20260501 14:02:06.294291 3715 debug-util.cc:398] Leaking SignalData structure 0x7b080010f8e0 after lost signal to thread 3718
I20260501 14:02:06.573400 5983 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:06.574514 5983 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:06.577185 5983 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:06.578953 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:02:06.579876 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:02:06.580235 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:06.581038 5983 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:06.587334 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 204800ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:06.693707 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:06.703070 5990 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:06.703564 5990 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:06.706489 5990 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:06.707643 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:02:06.708216 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:02:06.708537 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:06.709219 5990 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:06.836781 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 204800ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:07.281455 5993 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:07.281986 5993 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:07.284158 5993 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:07.285105 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:07.285523 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:07.287111 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:07.287840 5993 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:07.290020 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 205500ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:07.544595 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 205500ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:07.800505 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:07.810431 6001 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:07.811671 6001 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:07.827718 6001 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:07.829267 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:07.830353 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:07.832052 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:07.833012 6001 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:08.037822 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 206208ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:08.281257 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 206208ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:08.433568 6003 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:08.434208 6003 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:08.436311 6003 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:08.436940 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:08.437289 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:08.438393 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:08.439003 6003 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:08.732594 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 206944ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:08.808346 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:08.945464 6007 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:08.945897 6007 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:08.947505 6007 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:08.949373 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:02:08.950668 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:02:08.951059 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:08.951784 6007 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:08.992117 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 206944ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:09.053167 6008 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:09.053695 6008 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:09.055775 6008 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:09.056790 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:09.057062 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:09.058820 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:09.059497 6008 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:09.436086 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 207655ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:09.673241 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 207655ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:09.764772 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:10.088037 6015 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:10.089021 6015 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:10.092682 6015 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:10.093364 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:10.095299 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:10.096750 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:10.097623 6015 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:10.109910 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 208336ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:10.338025 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 208336ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:10.538378 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:10.711475 6018 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:10.712013 6018 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:10.713936 6018 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:10.715453 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:02:10.715993 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:02:10.716321 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:10.716915 6018 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:10.790076 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 209001ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:11.031705 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 209001ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:11.099503 6020 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:11.100036 6020 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:11.102298 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:11.103185 6020 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:11.104223 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:11.105512 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:11.106174 6020 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:11.478785 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 209694ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:11.699699 6024 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:11.700246 6024 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:11.701900 6024 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:11.702840 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:11.703233 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:11.704589 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:11.716678 6024 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:11.717883 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:11.724083 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 209694ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:12.163472 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 210387ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:12.412456 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 210387ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:12.760418 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:12.764905 6034 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:12.765383 6034 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:12.768945 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:12.768838 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:12.770730 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:12.772316 6035 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:02:12.767882 6034 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
W20260501 14:02:12.893702 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 211075ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:13.136910 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 211075ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:13.583240 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 211800ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:13.598938 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:13.698315 6041 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:13.698779 6041 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:13.700800 6041 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:13.701476 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:13.701776 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:13.703081 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:13.703759 6041 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:13.854403 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 211800ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:14.252610 6044 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:14.253028 6044 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:14.254489 6044 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:14.255296 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:14.255722 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:14.257200 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:14.257922 6044 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:14.302261 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 212517ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:14.433461 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:14.434839 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 111: this message will repeat every 5th retry.
W20260501 14:02:14.555727 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 212517ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:14.668553 3139 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.1.16.188:38253 timed out after 30.000s (SENT)
I20260501 14:02:14.816047 6050 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:14.816540 6050 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:14.818370 6050 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:14.820905 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:14.821208 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:14.822688 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:14.823662 6050 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:15.026404 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 213219ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:15.143692 3143 maintenance_manager.cc:419] P 0a1d2cd221924677a3ef81b081e5e016: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033380
I20260501 14:02:15.184293 3142 maintenance_manager.cc:643] P 0a1d2cd221924677a3ef81b081e5e016: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.033s sys 0.004s Metrics: {"bytes_written":5312,"cfile_init":1,"dirs.queue_time_us":944,"dirs.run_cpu_time_us":297,"dirs.run_wall_time_us":1054,"drs_written":1,"lbm_read_time_us":251,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2424,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":2,"thread_start_us":1270,"threads_started":2,"wal-append.queue_time_us":960}
I20260501 14:02:15.187402 3143 maintenance_manager.cc:419] P 0a1d2cd221924677a3ef81b081e5e016: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.232169
W20260501 14:02:15.281806 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 213219ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:15.295125 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:15.302006 3142 maintenance_manager.cc:643] P 0a1d2cd221924677a3ef81b081e5e016: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.114s user 0.109s sys 0.001s Metrics: {"bytes_written":38584,"cfile_cache_hit":16,"cfile_cache_hit_bytes":32052,"cfile_cache_miss":30,"cfile_cache_miss_bytes":46158,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":2104,"dirs.run_cpu_time_us":1369,"dirs.run_wall_time_us":4159,"drs_written":1,"lbm_read_time_us":3104,"lbm_reads_lt_1ms":62,"lbm_write_time_us":2866,"lbm_writes_lt_1ms":31,"mutex_wait_us":220,"num_input_rowsets":2,"peak_mem_usage":83698,"rows_written":100,"spinlock_wait_cycles":22400,"thread_start_us":2093,"threads_started":3}
I20260501 14:02:15.305191 3143 maintenance_manager.cc:419] P 0a1d2cd221924677a3ef81b081e5e016: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7670 bytes on disk
I20260501 14:02:15.306608 3142 maintenance_manager.cc:643] P 0a1d2cd221924677a3ef81b081e5e016: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":180,"lbm_reads_lt_1ms":4}
W20260501 14:02:15.700953 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 213945ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:15.744747 6047 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:15.745246 6047 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:15.747349 6047 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:15.751976 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:02:15.752568 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:02:15.752903 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:15.753832 6047 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 14:02:15.844048 6061 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:15.844789 6061 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:15.847571 6061 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:15.848112 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:15.849262 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:15.850823 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:15.851833 6061 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:15.856472 4617 debug-util.cc:398] Leaking SignalData structure 0x7b0800074fa0 after lost signal to thread 4319
W20260501 14:02:15.857307 4617 debug-util.cc:398] Leaking SignalData structure 0x7b080009d980 after lost signal to thread 4620
W20260501 14:02:15.949925 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 213945ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:16.299144 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:16.394825 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 214613ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:16.634939 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 214613ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:17.086668 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 215298ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:17.282224 6070 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:17.282742 6070 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:17.284679 6070 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:17.285626 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:17.285730 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:17.287392 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:17.288156 6070 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:17.335477 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 215298ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:17.467880 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:17.788070 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 215998ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:17.927726 6073 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:17.928395 6073 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:17.931313 6073 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:17.932979 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:02:17.933584 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:02:17.933852 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:17.934412 6073 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:18.034423 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 215998ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:18.471800 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 216697ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:18.512454 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:18.700964 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 216697ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:19.145428 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 217364ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:19.194633 6079 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:19.195508 6079 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:19.197052 6079 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:19.198093 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:19.198503 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:19.201134 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:19.201798 6079 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:19.402096 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 217364ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:19.728161 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:19.765045 6086 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:19.765520 6086 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:19.768462 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:19.769755 6086 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:19.770862 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:19.772135 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:19.772827 6086 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:19.849768 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 218065ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:20.102568 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 218065ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:20.280977 3557 debug-util.cc:398] Leaking SignalData structure 0x7b08000f2d80 after lost signal to thread 3422
W20260501 14:02:20.282099 3557 debug-util.cc:398] Leaking SignalData structure 0x7b08001220a0 after lost signal to thread 3560
W20260501 14:02:20.579268 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 218766ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:20.829780 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 218766ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:20.969727 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:21.270159 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 219492ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:21.516716 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 219492ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:21.957283 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 220180ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:22.078034 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:22.204820 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 220180ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:22.524431 6097 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:22.524959 6097 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:22.526535 6097 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:22.527289 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:22.527875 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:22.529285 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:22.529969 6097 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:22.644527 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 220868ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:22.886623 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:22.894336 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 220868ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:23.046418 6103 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:23.046914 6103 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:23.048820 6103 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:23.049430 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:23.049746 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:23.052349 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:23.053015 6103 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:02:23.257869 6093 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:23.258404 6093 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:23.260313 6093 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:23.261931 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:02:23.262859 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:02:23.263234 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:23.263845 6093 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:23.347105 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 221557ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:23.600595 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 221557ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:23.619148 6107 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:23.619634 6107 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:23.621520 6107 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:23.622364 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:23.622566 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:23.623910 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:23.624589 6107 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:23.929538 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:24.030136 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 222264ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:24.281069 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 222264ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:24.693885 6113 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:24.694454 6113 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:24.697042 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:24.697147 6113 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:24.697830 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:24.699386 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:24.700032 6113 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:24.723260 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 222944ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:24.780921 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 116: this message will repeat every 5th retry.
W20260501 14:02:24.782902 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:24.964430 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 222944ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:25.372802 6110 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:25.373275 6110 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:25.375164 6110 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:25.378144 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:02:25.378791 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:02:25.379135 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:25.379774 6110 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:25.455837 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 223627ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:25.691603 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 223627ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:25.871793 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:26.083333 3561 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (220 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 14:02:26.143285 3719 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (153 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 14:02:26.145994 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 224355ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:26.391606 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 224355ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:26.601171 6124 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:26.601673 6124 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:26.604137 6124 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:26.604106 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:26.606326 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:26.608022 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:26.608605 6124 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:26.850915 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 225054ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:27.051581 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:27.105964 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 225054ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:27.569423 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 225769ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:27.756253 3867 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (154 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 14:02:27.815333 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 225769ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:28.087188 6129 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:28.087915 6129 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:28.090520 6129 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:28.091147 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:28.091445 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:28.093051 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:28.094470 6129 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:28.231515 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:28.257865 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 226478ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:28.506948 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 226478ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:28.637995 6134 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:28.638501 6134 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:28.640417 6134 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:28.641505 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:28.641909 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:28.643054 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:28.643713 6134 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:28.949425 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 227170ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:29.195111 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 227170ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:29.369300 6132 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:29.370185 6132 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:29.373251 6132 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:29.375838 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:02:29.377709 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:02:29.378072 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:29.378916 6138 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:29.388053 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:29.598577 6139 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:29.599190 6139 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:29.601344 6139 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:29.602979 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:29.602435 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:29.607411 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:29.608009 6139 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:29.642225 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 227858ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:29.894560 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 227858ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:30.336418 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 228557ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:30.557226 6143 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:30.557734 6143 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:30.559634 6143 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:30.560566 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:30.560600 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:30.561969 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:30.562873 6143 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:30.589843 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 228557ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:30.593060 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:31.029887 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 229253ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:31.271489 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 229253ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:31.648347 6138 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:31.648893 6138 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:31.651643 6138 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:31.653726 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:02:31.654337 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:02:31.654843 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:31.655474 6138 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:31.701773 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 229934ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:31.836444 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:31.947834 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 229934ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:32.402104 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 230610ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:32.635389 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 230610ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:32.772936 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:33.072755 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 231298ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:33.264603 6156 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:33.265066 6156 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:33.266976 6156 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:33.267724 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:33.267918 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:33.269243 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:33.269891 6156 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:33.317394 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 231298ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:33.758951 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 231980ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:33.799636 6159 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:33.800240 6159 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:33.802714 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:33.803260 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:33.804804 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:33.810410 6159 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:33.810933 6159 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:33.993956 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:34.006410 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 231980ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:34.155776 3185 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1ee0 after lost signal to thread 3122
W20260501 14:02:34.156488 3185 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9280 after lost signal to thread 3188
W20260501 14:02:34.462955 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 232669ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:34.713879 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 232669ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:34.901391 6164 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:34.902370 6164 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:34.904522 6164 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:34.905278 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:34.905495 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:34.906895 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:34.907547 6164 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:35.155000 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 233377ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:35.228868 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:35.414583 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 233377ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:35.422047 6170 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:35.422434 6170 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:35.424012 6170 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:35.424757 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:35.426831 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:35.428411 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:35.429128 6170 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:35.874258 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 234078ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x47c8f5 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x480c2d __interceptor_memset at ??:?
@ 0x7f493f7d8fd3 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260501 14:02:35.967015 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 121: this message will repeat every 5th retry.
W20260501 14:02:35.969550 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:36.155644 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 234078ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:36.591737 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 234819ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:36.830711 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 234819ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:36.894503 6180 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:36.895442 6180 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:36.897666 6180 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:36.898633 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:36.899089 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:36.900692 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:36.901710 6180 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:37.216496 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:37.282599 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 235494ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:37.435674 6186 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:37.436168 6186 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:37.438041 6186 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:37.439128 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:37.441776 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:37.442993 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:37.443812 6186 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:37.636776 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 235494ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:37.973996 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:38.023785 6190 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:38.024293 6190 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:38.026371 6190 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:38.027395 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:38.027395 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:38.028877 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:38.029546 6190 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:38.080230 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 236300ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:38.331280 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 236300ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:38.401353 4110 debug-util.cc:398] Leaking SignalData structure 0x7b080007fda0 after lost signal to thread 3931
W20260501 14:02:38.402060 4110 debug-util.cc:398] Leaking SignalData structure 0x7b0800075f40 after lost signal to thread 4114
I20260501 14:02:38.588786 6192 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:38.589306 6192 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:38.591249 6192 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:38.592257 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:38.592307 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:38.593752 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:38.594571 6192 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:38.768832 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 236994ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:39.017660 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 236994ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:39.170972 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:39.472702 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 237681ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:39.719084 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 237681ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:40.106364 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:40.157025 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 238382ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:40.401324 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 238382ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:40.508069 6204 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:40.508621 6204 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:40.510630 6204 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:40.511559 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:40.511557 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:40.513118 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:40.513789 6204 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:40.673578 4038 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b9f3b4229594e13959fec0aa8281c03 consensus state has no leader
W20260501 14:02:40.841562 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 239064ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:40.850147 6202 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:40.850823 6202 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:40.853415 6202 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:40.855013 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:02:40.855876 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:02:40.856220 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:40.857136 6202 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 14:02:41.061890 6208 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:41.062454 6208 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:41.071481 6208 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:41.072130 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:41.073861 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:41.075117 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:41.076745 6208 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:41.081996 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 239064ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:41.213071 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:41.535328 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 239745ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:41.565678 3878 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.16.188:38253)
I20260501 14:02:41.572336 4064 master_service.cc:954] ReplaceTablet: received request to replace tablet 151c6fb3afbd46a9a4c9cac128ad4ec0 from {username='slave'} at 127.0.0.1:35438
I20260501 14:02:41.626052 6214 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:41.626737 6214 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:41.628978 6214 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:41.629949 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:41.629962 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:41.631986 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:41.632683 6214 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:41.778431 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 239745ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:42.025943 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:42.207244 6219 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:42.207795 6219 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:42.210556 6219 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:42.211972 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:42.211969 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:42.213544 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:42.214598 6219 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:42.235316 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 240441ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:42.497563 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 240441ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:42.645154 6218 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:42.645658 6218 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:42.648327 6218 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:42.649155 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:02:42.661134 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:02:42.661478 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:42.662209 6218 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 14:02:42.784237 6223 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:42.784727 6223 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:42.786746 6223 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:42.787801 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:42.788259 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:42.791127 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:42.791772 6223 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:42.941244 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 241160ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:43.131299 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:43.195315 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 241160ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:43.420761 6227 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:43.426337 6227 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:43.429665 6227 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:43.430662 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:43.430662 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:43.432508 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:43.433256 6227 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:02:43.437814 6225 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:43.438292 6225 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:43.440199 6225 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:43.442082 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:02:43.442699 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:02:43.443032 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:43.443647 6225 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:43.651470 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 241858ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:43.884508 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 241858ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:43.986670 6230 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:43.987076 6230 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:43.989008 6230 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:43.989684 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:43.989897 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:43.991238 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:43.991760 6230 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:44.084241 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:44.335536 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 242547ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:44.533445 6234 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:44.534025 6234 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:44.536562 6234 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:44.537454 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:44.537904 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:44.539145 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:44.539820 6234 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:44.590188 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 242547ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:44.914178 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:45.033456 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 243253ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:45.135566 6239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:45.136873 6239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:45.140177 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:45.141290 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:45.143297 6239 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:45.143388 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:45.143988 6239 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:45.284363 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 243253ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:45.725711 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 243947ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:45.896229 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 126: this message will repeat every 5th retry.
W20260501 14:02:45.911904 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:45.968252 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 243947ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:46.418728 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 244631ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:46.437799 6245 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:46.438588 6245 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:46.440907 6245 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:46.442682 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:46.443691 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:46.445392 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:46.446507 6245 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:46.658072 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 244631ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:47.105825 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 245321ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:47.146782 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:47.372596 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 245321ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:47.419838 6252 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:47.420540 6252 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:47.422602 6252 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:47.423537 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:47.423982 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:47.425175 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:47.425859 6252 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:47.812422 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 246036ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:48.000082 6256 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:48.000720 6256 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:48.002615 6256 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:48.003346 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:48.005273 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:48.006582 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:48.007367 6256 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:48.061980 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 246035ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:48.182255 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:48.532670 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 246725ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:48.778350 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 246725ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:48.938861 6261 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:48.939255 6261 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:48.941156 6261 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:48.941856 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:48.942054 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:48.943674 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:48.944350 6261 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:49.232482 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 247441ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:49.256348 6260 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:49.256995 6260 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:49.259238 6260 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:49.260829 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:02:49.261483 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:02:49.261821 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:49.262527 6260 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:49.365489 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:49.470798 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 247441ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:49.630201 6258 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:49.630729 6258 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:49.632616 6258 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:49.633769 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:02:49.634305 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:02:49.634579 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:49.635140 6258 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:49.885864 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 248134ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:49.960697 6267 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:49.961215 6267 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:49.963101 6267 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:49.963982 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:49.964773 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:49.966373 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:49.967079 6267 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:50.112679 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 248134ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:50.175490 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:50.491247 6271 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:50.491791 6271 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:50.493752 6271 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:50.494837 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:50.495322 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:50.496495 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:50.497257 6271 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:50.576496 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 248777ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:50.818609 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 248777ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:50.952260 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:51.022902 6276 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:51.023440 6276 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:51.025380 6276 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:51.025928 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:51.026315 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:51.027642 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:51.028299 6276 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:51.272866 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 249481ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x47c8f5 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x4959cf __interceptor_pthread_setcancelstate at ??:?
@ 0x7f493f7d8fc0 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260501 14:02:51.531757 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 249481ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:51.921198 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:51.985476 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 250195ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:52.047703 6284 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:52.048255 6284 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:52.051019 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:52.051020 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:52.050089 6284 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:52.052685 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:52.054373 6284 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:52.235282 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 250195ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:52.670364 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 250899ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:52.887061 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 250899ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:52.926028 6280 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:52.926548 6280 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:52.928460 6280 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:52.930269 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:02:52.930889 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:02:52.931143 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:52.931643 6280 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:53.152693 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:53.348207 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 251550ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:53.587380 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 251550ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:53.924531 6291 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:53.925055 6291 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:53.927731 6291 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:53.928735 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:53.929538 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:53.931458 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:53.932124 6291 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:54.032395 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 252250ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:54.280668 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 252250ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:54.294952 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:54.702325 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 252944ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:54.945633 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 252944ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:55.177352 6296 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:55.177970 6296 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:55.179956 6296 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:55.182279 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:02:55.182929 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:02:55.183362 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:55.184415 6296 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:55.393220 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 253608ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x47c8f5 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x4959cf __interceptor_pthread_setcancelstate at ??:?
@ 0x7f493f7d8fc0 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
I20260501 14:02:55.399102 6299 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:55.399737 6299 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:55.402334 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:55.403407 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:55.404603 6299 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:55.404927 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:55.405840 6299 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:55.477649 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:55.673636 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 253608ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:55.996631 6296 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:02:55.997149 6296 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:02:55.999577 6296 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:02:56.001749 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:02:56.002328 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:02:56.002626 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:02:56.003130 6296 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:02:56.126318 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 254337ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:56.378684 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 254337ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:56.398082 6306 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:56.398939 6306 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:56.401232 6306 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:56.402004 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:56.403234 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:56.405283 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:56.406136 6306 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:56.493909 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 131: this message will repeat every 5th retry.
W20260501 14:02:56.501317 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:56.822381 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 255042ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:57.004639 6311 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:57.005146 6311 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:57.007933 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:57.008951 6311 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:57.009585 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:57.010957 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:57.013082 6312 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:57.057502 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 255042ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:57.338639 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:57.497653 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 255720ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:57.539613 6315 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:57.540077 6315 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:57.541917 6315 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:57.542785 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:57.542927 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:57.544507 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:57.545166 6315 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:57.737867 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 255720ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:58.183537 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 256401ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:58.238504 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:02:58.451552 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 256401ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:58.535997 6318 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:58.536482 6318 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:58.539080 6318 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:58.539990 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:58.540059 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:58.541805 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:58.542346 6318 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:58.889907 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 257114ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:59.068768 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:02:59.074605 6323 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:59.075176 6323 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:59.083987 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:59.089329 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:59.090756 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:59.090590 6323 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:59.091842 6323 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:59.132881 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 257114ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:02:59.594014 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 257796ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:02:59.600589 6325 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:02:59.601094 6325 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:02:59.603060 6325 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:02:59.604149 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:02:59.604100 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:02:59.605921 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:02:59.606593 6325 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:02:59.821501 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 257796ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:00.113830 6330 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:00.114810 6330 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:00.119550 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:00.120800 6330 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:00.122826 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:00.125244 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:00.126788 6330 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:00.251245 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:00.268216 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 258484ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:00.524899 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 258484ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:00.631067 6334 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:00.631578 6334 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:00.633211 6334 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:00.634248 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:00.634166 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:00.635654 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:00.636606 6334 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:00.981693 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 259188ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:01.030809 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:01.236160 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 259188ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:01.623732 6340 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:01.624344 6340 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:01.626293 6340 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:01.627290 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:01.629707 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:01.631166 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:01.632148 6340 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:01.694255 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 259899ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:01.944957 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 259899ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:02.142088 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:02.401546 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 260608ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:02.560570 6342 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:03:02.561295 6342 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:03:02.563476 6342 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:03:02.565176 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:03:02.565959 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:03:02.566311 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:03:02.567022 6342 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:03:02.644933 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 260608ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:03.091325 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 261308ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:03.182951 4617 debug-util.cc:398] Leaking SignalData structure 0x7b080009d9a0 after lost signal to thread 4319
W20260501 14:03:03.183646 4617 debug-util.cc:398] Leaking SignalData structure 0x7b080006f040 after lost signal to thread 4620
W20260501 14:03:03.356858 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 261308ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:03.366972 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:03.804976 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 262020ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:04.047137 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 262020ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:04.398634 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:04.533666 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 262710ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:04.771467 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 262710ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:05.222764 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 263434ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:05.233192 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:05.478346 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 263434ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:05.912815 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 264142ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:06.159265 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 264142ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:06.165509 6358 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:06.166186 6358 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:06.168053 6358 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:06.169100 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:06.170222 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:06.172071 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:06.172631 6358 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:06.223824 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 136: this message will repeat every 5th retry.
W20260501 14:03:06.229034 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:06.618367 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 264822ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:06.861480 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 264822ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:07.329692 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 265524ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:07.417064 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:07.588200 6367 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
W20260501 14:03:07.588785 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 265524ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:07.588729 6367 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:07.590737 6367 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:07.591781 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:07.592362 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:07.593580 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:07.594216 6367 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:08.024200 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 266252ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:08.186959 6369 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:08.187464 6369 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:08.190632 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:08.195339 6369 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:08.196352 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:08.197733 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:08.198922 6369 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:08.277379 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 266252ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:08.632519 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:08.725054 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 266940ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:08.724993 6373 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:08.725471 6373 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:08.727865 6373 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:08.731520 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:08.732040 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:08.733621 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:08.734632 6373 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:08.964948 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 266940ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:09.425655 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 267628ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:09.673882 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 267628ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:09.717082 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:10.083258 6380 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:10.083838 6380 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:10.085860 6380 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:10.087486 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:10.089488 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:10.090970 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:10.091710 6380 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:10.106803 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 268337ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:10.358170 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 268337ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:10.708650 6383 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:10.709038 6383 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:10.710608 6383 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:10.711227 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:10.711516 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:10.712774 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:10.713301 6383 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:10.797734 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 269021ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x47c8f5 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x480c2d __interceptor_memset at ??:?
@ 0x7f493f7d8fd3 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260501 14:03:10.859349 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:11.042825 6377 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260501 14:03:11.043421 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 269021ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:11.043471 6377 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:03:11.046013 6377 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:03:11.047789 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:03:11.048441 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:03:11.048774 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:03:11.049481 6377 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 14:03:11.229876 6386 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:11.230489 6386 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:11.232915 6386 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:11.233140 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:11.233407 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:11.234781 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:11.236027 6386 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:11.498078 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 269706ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:11.619419 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:11.683198 6389 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:03:11.683706 6389 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:03:11.685704 6389 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:03:11.688674 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:03:11.689286 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:03:11.689623 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:03:11.690441 6389 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:03:11.742977 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 269706ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:12.190768 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 270406ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:12.439568 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 270406ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:12.462180 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:12.736243 6396 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:12.736717 6396 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:12.738571 6396 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:12.739533 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:12.739828 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:12.741039 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:12.741526 6396 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:12.871208 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 271103ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:13.094005 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 271103ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x4d1295 __tsan_atomic32_compare_exchange_strong at ??:?
@ 0x7f493f7fe832 base::subtle::NoBarrier_CompareAndSwap() at ??:0
@ 0x7f493f7fe792 kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x5b3bc5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5b39fa std::__1::function<>::operator()() at ??:?
I20260501 14:03:13.253115 6403 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:13.253610 6403 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:13.255131 6403 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:13.255966 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:13.255983 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:13.257153 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:13.257714 6403 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:13.469907 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:13.547062 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 271757ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:13.787741 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 271757ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:14.264901 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 272451ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:14.275300 6410 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:14.275786 6410 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:14.278802 6410 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:14.280105 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:14.281603 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:14.283106 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:14.283933 6410 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:14.521507 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 272451ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:14.605086 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:14.693770 3139 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.1.16.188:38253 timed out after 30.000s (SENT)
W20260501 14:03:14.826249 3557 debug-util.cc:398] Leaking SignalData structure 0x7b0800110f80 after lost signal to thread 3422
W20260501 14:03:14.827421 3557 debug-util.cc:398] Leaking SignalData structure 0x7b08000fc360 after lost signal to thread 3560
W20260501 14:03:14.964439 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 273184ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:15.193367 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 273184ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:15.353106 6418 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:15.354005 6418 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:15.358878 6418 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:15.359401 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:15.361624 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:15.363161 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:15.364076 6418 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:15.376487 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:15.598271 6419 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:03:15.598834 6419 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:03:15.600781 6419 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:03:15.602949 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:03:15.603554 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:03:15.603881 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:03:15.604549 6419 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:03:15.647792 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 273856ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:15.897406 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 273856ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:16.323503 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 141: this message will repeat every 5th retry.
W20260501 14:03:16.329849 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:16.340003 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 274561ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:16.354557 6427 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:16.355315 6427 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:16.357681 6427 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:16.358307 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:16.360808 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:16.363318 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:16.364498 6427 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:16.584388 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 274561ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:17.041657 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 275247ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:17.294723 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 275247ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:17.541697 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:17.730046 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 275958ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:17.740906 6436 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:17.741567 6436 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:17.743542 6436 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:17.744491 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:17.745643 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:17.746965 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:17.747632 6436 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:17.974674 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 275958ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:18.337275 6439 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:18.338186 6439 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:18.340853 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:18.340095 6439 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:18.341547 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:18.343444 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:18.344108 6439 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:18.428759 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 276638ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:18.678072 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 276638ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:18.721002 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:18.888684 6444 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:18.889261 6444 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:18.891426 6444 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:18.892362 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:18.892870 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:18.894543 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:18.895265 6444 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:19.108541 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 277341ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:19.371798 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 277341ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x47c644 __tsan::ProcessPendingSignals() at ??:?
@ 0x49e580 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7f493f052c42 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7f493f0518de _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x5b3bc5 std::__1::__function::__value_func<>::operator()() at ??:?
I20260501 14:03:19.570896 6442 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:03:19.571449 6442 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:03:19.573129 6442 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:03:19.574898 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:03:19.580796 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:03:19.581161 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:03:19.582285 6442 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:03:19.807574 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 278035ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:19.904788 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:20.044041 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 278035ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:20.501324 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 278707ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:20.736354 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 278707ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:20.935818 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:21.108690 6457 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:21.109287 6457 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:21.112465 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:21.115605 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:21.117208 6457 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:21.117128 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:21.117892 6457 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:21.176117 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 279399ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:21.430152 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 279399ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:21.813199 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:21.868584 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 280093ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:22.114216 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 280093ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:22.560948 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 280777ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:22.685887 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:22.829627 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 280777ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:23.310398 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 281492ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:23.559621 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 281492ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:23.573238 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:24.013188 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 282223ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:24.270236 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 282223ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:24.532275 6467 raft_consensus.cc:493] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:03:24.532779 6467 raft_consensus.cc:515] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:03:24.534642 6467 leader_election.cc:290] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:03:24.551679 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
W20260501 14:03:24.552263 3448 leader_election.cc:343] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: a9ecb5b51ae04f32a4f07f828d563fa4
I20260501 14:03:24.552639 3448 leader_election.cc:304] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:03:24.553335 6467 raft_consensus.cc:2749] T a9ecb5b51ae04f32a4f07f828d563fa4 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:03:24.562429 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:24.722761 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 282933ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:24.967298 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 282933ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:25.419888 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 283630ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:25.435339 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:25.517470 6477 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:25.517961 6477 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:25.519977 6477 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:25.520700 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:25.521028 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:25.522612 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:25.523283 6477 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:25.675463 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 283630ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:26.135600 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 284339ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:26.244477 3719 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (193 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 14:03:26.395427 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 284339ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:26.514112 3715 debug-util.cc:398] Leaking SignalData structure 0x7b0800080660 after lost signal to thread 3572
W20260501 14:03:26.515334 3715 debug-util.cc:398] Leaking SignalData structure 0x7b080011d4c0 after lost signal to thread 3718
W20260501 14:03:26.524773 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 146: this message will repeat every 5th retry.
W20260501 14:03:26.532696 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:26.834180 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 285059ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:26.960264 3561 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (279 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 14:03:27.086820 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 285059ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:27.289911 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:27.552747 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 285754ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:27.788512 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 285754ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:27.815205 6488 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:27.815876 6488 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:27.819639 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:27.820573 6488 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:27.820801 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:27.822501 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:27.823256 6488 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:27.849309 3867 heartbeater.cc:646] Failed to heartbeat to 127.1.16.188:38253 (194 consecutive failures): Timed out: Failed to ping master at 127.1.16.188:38253: Ping RPC to 127.1.16.188:38253 timed out after 1.000s (SENT)
W20260501 14:03:28.092413 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:28.223973 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 286451ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:28.364871 6491 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:28.365396 6491 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:28.367332 6491 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:28.368309 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:28.368257 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:28.370520 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:28.371539 6491 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:28.469087 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 286451ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:28.906410 6494 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:28.907002 6494 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:28.909687 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:28.909829 6494 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:28.911417 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:28.912833 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:28.913800 6494 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:28.914371 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 287132ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:29.042128 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:29.155428 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 287132ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:29.524464 6501 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:29.525053 6501 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:29.530799 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:29.535950 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:29.538025 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:29.539819 6501 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:29.540625 6503 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:29.610803 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 287818ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:29.866690 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 287818ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:30.011632 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:30.373330 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 288530ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:30.617435 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 288530ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:30.792443 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:31.070709 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 289280ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:31.311106 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 289280ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:31.375478 4110 debug-util.cc:398] Leaking SignalData structure 0x7b080006f1c0 after lost signal to thread 3931
W20260501 14:03:31.376164 4110 debug-util.cc:398] Leaking SignalData structure 0x7b0800075de0 after lost signal to thread 4114
W20260501 14:03:31.615603 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:31.770852 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 289974ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:32.020277 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 289974ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:32.460920 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 290683ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:32.668651 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:32.714663 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 290683ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:32.761329 6521 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:32.761826 6521 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:32.763708 6521 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:32.764457 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:32.764621 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:32.766181 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:32.766886 6521 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260501 14:03:32.902822 6518 raft_consensus.cc:493] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:03:32.903537 6518 raft_consensus.cc:515] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:03:32.905936 6518 leader_election.cc:290] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:03:32.907673 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
W20260501 14:03:32.908528 3448 leader_election.cc:343] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: 6309ec735c424460ad5a74c6b7ec0d8a
I20260501 14:03:32.908880 3448 leader_election.cc:304] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:03:32.909972 6518 raft_consensus.cc:2749] T 6309ec735c424460ad5a74c6b7ec0d8a P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260501 14:03:33.172778 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 291378ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:33.431147 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 291377ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:33.607015 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:33.867384 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 292094ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:34.127092 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 292094ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:34.269510 6527 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:34.269994 6527 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:34.271771 6527 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:34.272630 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:34.274279 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:34.275499 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:34.276122 6527 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:34.573523 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 292790ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:34.606076 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:34.809543 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 292790ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:34.833814 6532 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:34.834322 6532 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:34.835911 6532 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:34.836673 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:34.836984 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:34.838258 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:34.840122 6532 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:35.276948 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 293473ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:35.396709 6534 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:35.397223 6534 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:35.399168 6534 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:35.399868 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:35.401393 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:35.403294 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:35.404124 6534 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:35.507968 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 293473ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:35.534039 3448 consensus_peers.cc:597] T 19f08ae176df4a369c635a562b3dc3ba P 9154033cea374a10a5225c53b8bcf122 -> Peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Couldn't send request to peer 6af0cedc7f3c47299b93e6e92d8e02d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f08ae176df4a369c635a562b3dc3ba. This is attempt 151: this message will repeat every 5th retry.
W20260501 14:03:35.539450 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:35.955111 6539 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:35.955825 6539 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:35.958041 6539 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:35.958758 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:35.959698 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:35.961642 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:35.962517 6539 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:35.974598 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 294171ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:36.231858 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 294171ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:36.381829 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:36.570075 6544 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:36.570686 6544 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:36.572718 6544 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:36.573473 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:36.574141 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:36.575835 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:36.577842 6544 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:36.678334 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 294895ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:36.917277 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 294895ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:37.177856 6550 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:37.178368 6550 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:37.180323 6550 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:37.181303 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:37.181318 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:37.183491 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:37.184327 6550 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:37.369513 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 295580ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:37.550189 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:37.638132 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 295580ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:38.077564 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 296301ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:38.328603 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 296301ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:38.607151 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:38.636417 6557 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:38.636906 6557 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:38.639686 6557 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:38.639637 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:38.639663 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:38.641000 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:38.642045 6557 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:38.791321 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 296991ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:39.033325 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 296991ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:39.209563 6555 raft_consensus.cc:493] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 14:03:39.210179 6555 raft_consensus.cc:515] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9154033cea374a10a5225c53b8bcf122" member_type: VOTER last_known_addr { host: "127.1.16.129" port: 44335 } } peers { permanent_uuid: "6af0cedc7f3c47299b93e6e92d8e02d1" member_type: VOTER last_known_addr { host: "127.1.16.130" port: 45857 } } peers { permanent_uuid: "5c239ccbdc554619a237d78e958ffbc2" member_type: VOTER last_known_addr { host: "127.1.16.131" port: 38685 } }
I20260501 14:03:39.212224 6555 leader_election.cc:290] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857), 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685)
W20260501 14:03:39.213986 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6af0cedc7f3c47299b93e6e92d8e02d1 (127.1.16.130:45857): Not found: Tablet not found: ba69864093504babbb00fb6170774778
W20260501 14:03:39.214670 3448 leader_election.cc:343] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c239ccbdc554619a237d78e958ffbc2 (127.1.16.131:38685): Not found: Tablet not found: ba69864093504babbb00fb6170774778
I20260501 14:03:39.214991 3448 leader_election.cc:304] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9154033cea374a10a5225c53b8bcf122; no voters: 5c239ccbdc554619a237d78e958ffbc2, 6af0cedc7f3c47299b93e6e92d8e02d1
I20260501 14:03:39.215657 6555 raft_consensus.cc:2749] T ba69864093504babbb00fb6170774778 P 9154033cea374a10a5225c53b8bcf122 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260501 14:03:39.257781 6559 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:39.258327 6559 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:39.260322 6559 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:39.261083 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:39.261551 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:39.262816 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:39.263474 6559 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:39.381731 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
W20260501 14:03:39.459782 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 297696ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:39.697590 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 297696ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:39.885708 6565 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:39.886780 6565 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:39.889091 6565 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:39.891254 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:39.893123 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:39.895067 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:39.896075 6565 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:40.148098 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 298361ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:40.394953 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 298361ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260501 14:03:40.445015 3154 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
I20260501 14:03:40.483624 6571 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:40.484122 6571 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:40.485993 6571 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:40.486941 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:40.487012 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:40.488541 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:40.490094 6571 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:40.700240 4038 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7b9f3b4229594e13959fec0aa8281c03 (table default.table_4db61a7562964779afbc5ff93b00a5f5 [id=b1334bc9986e4463add9b290ab6294ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b9f3b4229594e13959fec0aa8281c03 consensus state has no leader
W20260501 14:03:40.841408 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 299058ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:41.055475 6574 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a1d2cd221924677a3ef81b081e5e016)
I20260501 14:03:41.056037 6574 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a1d2cd221924677a3ef81b081e5e016" member_type: VOTER last_known_addr { host: "127.1.16.190" port: 38561 } } peers { permanent_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" member_type: VOTER last_known_addr { host: "127.1.16.189" port: 44255 } } peers { permanent_uuid: "6b0f8911dd5d431d9d8d3124a4456262" member_type: VOTER last_known_addr { host: "127.1.16.188" port: 38253 } }
I20260501 14:03:41.058800 3164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "0a1d2cd221924677a3ef81b081e5e016" is_pre_election: true
I20260501 14:03:41.058990 6574 leader_election.cc:290] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a1d2cd221924677a3ef81b081e5e016 (127.1.16.190:38561), 6b0f8911dd5d431d9d8d3124a4456262 (127.1.16.188:38253)
I20260501 14:03:41.059974 4080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15dd70d7c5494c3c911f0a780dc9a2af" candidate_term: 8 candidate_status { last_received { term: 7 index: 234 } } ignore_live_leader: false dest_uuid: "6b0f8911dd5d431d9d8d3124a4456262" is_pre_election: true
I20260501 14:03:41.061247 4502 leader_election.cc:304] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15dd70d7c5494c3c911f0a780dc9a2af; no voters: 0a1d2cd221924677a3ef81b081e5e016, 6b0f8911dd5d431d9d8d3124a4456262
I20260501 14:03:41.062285 6574 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15dd70d7c5494c3c911f0a780dc9a2af [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260501 14:03:41.085397 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 299058ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260501 14:03:41.347914 4046 maintenance_manager.cc:419] P 6b0f8911dd5d431d9d8d3124a4456262: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 6381 bytes on disk
I20260501 14:03:41.370097 4042 maintenance_manager.cc:643] P 6b0f8911dd5d431d9d8d3124a4456262: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.014s sys 0.000s Metrics: {"dirs.queue_time_us":1084,"dirs.run_cpu_time_us":327,"dirs.run_wall_time_us":330,"thread_start_us":7335,"threads_started":2,"wal-append.queue_time_us":1288}
W20260501 14:03:41.461597 3152 catalog_manager.cc:5504] Ignoring report from unknown tablet 19f08ae176df4a369c635a562b3dc3ba
F20260501 14:03:41.517697 3897 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet RPC to 127.1.16.188:38253 timed out after 59.943s (SENT)
I20260501 14:03:41.519735 4523 maintenance_manager.cc:419] P 15dd70d7c5494c3c911f0a780dc9a2af: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033357
*** Check failure stack trace: ***
*** Aborted at 1777644221 (unix time) try "date -d @1777644221" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000442) received by PID 1090 (TID 0x7faa1e5fe700) from PID 1090; stack trace: ***
W20260501 14:03:41.533982 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 299748ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x2de460 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e0943 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7faa34100980 (unknown) at ??:0
@ 0x7faa2f2c1e87 gsignal at ??:0
@ 0x7faa2f2c37f1 abort at ??:0
@ 0x2e4456 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20260501 14:03:41.565706 4521 maintenance_manager.cc:643] P 15dd70d7c5494c3c911f0a780dc9a2af: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.045s user 0.011s sys 0.009s Metrics: {"bytes_written":764,"delete_count":2,"lbm_write_time_us":446,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":10156,"threads_started":1,"update_count":6,"wal-append.queue_time_us":9825}
I20260501 14:03:41.567737 4523 maintenance_manager.cc:419] P 15dd70d7c5494c3c911f0a780dc9a2af: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
@ 0x7faa32359da4 google::LogMessage::SendToLog() at ??:0
@ 0x7faa3235a950 google::LogMessage::Flush() at ??:0
@ 0x7faa3235fa8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x361192 kudu::MasterStressTest::ReplaceTabletThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38e45e _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846
@ 0x38e311 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38da5b std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2de320 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7faa340f56db start_thread at ??:0
@ 0x7faa2f3a461f clone at ??:0
I20260501 14:03:41.641368 4521 maintenance_manager.cc:643] P 15dd70d7c5494c3c911f0a780dc9a2af: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.073s user 0.026s sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":705,"cfile_cache_miss":2,"cfile_cache_miss_bytes":663,"cfile_init":1,"delete_count":68,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":10034,"lbm_read_time_us":796,"lbm_reads_lt_1ms":6,"lbm_write_time_us":1116,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":9837,"threads_started":1,"update_count":6}
W20260501 14:03:41.841696 3952 kernel_stack_watchdog.cc:198] Thread 4037 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 299748ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x479650 __tsan::CallUserSignalHandler() at ??:?
@ 0x47bc38 rtl_sigaction() at ??:?
@ 0x7f4937917980 (unknown) at ??:0
@ 0x7f4937916d50 __GI___nanosleep at ??:0
@ 0x49e5a4 __interceptor_nanosleep at ??:?
@ 0x7f493f7d8fe5 boost::core::sp_thread_sleep() at ??:0
@ 0x7f493f7d8f8f boost::detail::yield() at ??:0
@ 0x7f493f7fe76c kudu::rw_semaphore::lock() at ??:0
@ 0x7f493f7fe6fa kudu::rw_spinlock::lock() at ??:0
@ 0x7f493f025a49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7f493eff974e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7f493eff85c4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7f493eff903a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f493f052d62 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f493f052cf2 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f493f052c82 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0